Skip to content
Implementation Science at UW

Keywords: Measurement and Measure Development

✪ Testing Three Alternate Methods to Direct Observation in Measuring Use of Discrete Youth Cognitive Behavioral Techniques: A Secondary Analysis

Authors:

Simone H Schriger, Steven C Marcus, Emily M Becker-Haimes, Shannon Dorsey, David S Mandell, Bryce D McLeod, Sonja K Schoenwald & Rinad S Beidas

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: September 2025

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Cognitive behavioral therapy (CBT), an umbrella term for therapeutic techniques guided by cognitive behavioral theory, is an evidence-based approach for many psychiatric conditions in youth. A stronger dose of CBT delivery is thought to improve youth clinical outcomes. While a critical indicator of care quality, measuring the use of CBT techniques feasibly and affordably is challenging. Certain CBT techniques (e.g., more concrete and observable) may be easier to measure than others using low-cost methods, such as clinician self-report; however, this has not been studied.

Method

To assess the concordance of three methods of measuring CBT technique use with direct observation (DO), clinicians from 27 community agencies (n = 126; Mage = 37.7 years, SD = 12.8; 76% female) were randomized 1:1:1 to a self-report, chart-stimulated recall (CSR; semistructured interviews with the chart available), or behavioral rehearsal (BR; simulated role-plays) condition. In previous work using a global score aggregating 12 CBT techniques, only BR produced scores that did not differ from DO. This secondary analysis examined the concordance of these alternate methods with DO for each discrete CBT technique, testing for differential concordance across cognitive techniques (e.g., cognitive education) compared to behavioral techniques (e.g., behavioral activation).

Results

Results of three-level mixed effects regression models indicated that BR scores did not differ significantly from DO for any techniques, and for nine techniques, neither did CSR (all ps > .05). Contrastingly, self-report scores differed from DO for all but one technique, with greater concordance for behavioral than cognitive techniques (z = −3.29, p < .001).

Conclusions

Unlike previous findings using an aggregate score, we found that both BR and CSR did not differ significantly from DO for most techniques tested. These findings have implications within implementation research and usual care settings; they support multiple viable measurement methods that are less resource-intensive than DO.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): a modified Delphi study

Authors:

Stephanie K. Brewer, Catherine M. Corbin, Ana A. Baumann, Shannon Wiltsey Stirman, Janine M. Jones, Michael D. Pullmann, Aaron R. Lyon & MODIFI Expert Panel

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: June 2024

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

Method

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability and inclusion (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

Results

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

Discussion

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

Contributions to the literature
  • Adaptation of evidence-based practices, programs, and treatments is often needed to ensure successful implementation. However, existing theories/models/frameworks do not outline concrete adaptation steps that are feasible for real-world settings.
  • We created a resource-efficient method (MODIFI) for developing and assessing intervention adaptations in local settings. This method was iteratively developed through a mixed-methods modified Delphi process, which resulted in consensus among experts in implementation science and practice.
  • MODIFI leverages human centered design techniques and is tailored to the needs of intervention implementers in applied settings, filling a crucial gap in the research to practice pipeline.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): a modified Delphi study

Authors:

Stephanie K. Brewer, Catherine M. Corbin, Ana A. Baumann, Shannon Wiltsey Stirman, Janine M. Jones, Michael D. Pullmann, Aaron R. Lyon & MODIFI Expert Panel

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: June 2024

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

Method

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability and inclusion (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

Results

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

Discussion

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

Contributions to the literature
  • Adaptation of evidence-based practices, programs, and treatments is often needed to ensure successful implementation. However, existing theories/models/frameworks do not outline concrete adaptation steps that are feasible for real-world settings.
  • We created a resource-efficient method (MODIFI) for developing and assessing intervention adaptations in local settings. This method was iteratively developed through a mixed-methods modified Delphi process, which resulted in consensus among experts in implementation science and practice.
  • MODIFI leverages human centered design techniques and is tailored to the needs of intervention implementers in applied settings, filling a crucial gap in the research to practice pipeline.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**