Aaron R. Lyon Archives - Page 3 of 6 - Implementation Science at UW

✪ A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers

Abstract:

Introduction

Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms.

Method

Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools–Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure.

Results

Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested.

Conclusion

The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Protocol for a hybrid type 3 effectiveness-implementation trial of a pragmatic individual-level implementation strategy for supporting school-based prevention programming

Abstract:

Background

For approximately one in five children who have social, emotional, and behavioral (SEB) challenges, accessible evidence-based prevention practices (EBPPs) are critical. In the USA, schools are the primary setting for children’s SEB service delivery. Still, EBPPs are rarely adopted and implemented by front-line educators (e.g., teachers) with sufficient fidelity to see effects. Given that individual behavior change is ultimately required for successful implementation, focusing on individual-level processes holds promise as a parsimonious approach to enhance impact.

Beliefs and Attitudes for Successful Implementation in Schools for Teachers (BASIS-T) is a pragmatic, multifaceted pre-implementation strategy targeting volitional and motivational mechanisms of educators’ behavior change to enhance implementation and student SEB outcomes. This study protocol describes a hybrid type 3 effectiveness-implementation trial designed to evaluate the main effects, mediators, and moderators of the BASIS-T implementation strategy as applied to Positive Greetings at the Door, a universal school-based EBPP previously demonstrated to reduce student disruptive behavior and increase academic engagement.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Core Components and Implementation Determinants of Multilevel Service Delivery Frameworks Across Child Mental Health Service Settings

Abstract:

Multilevel service delivery frameworks are approaches to structuring and organizing a spectrum of evidence-based services and supports, focused on assessment, prevention, and intervention designed for the local context. Exemplar frameworks in child mental health include positive behavioral interventions and supports in education, collaborative care in primary care, and systems of care in community mental health settings. Yet, their high-quality implementation has lagged. This work proposes a conceptual foundation for multilevel service delivery frameworks spanning diverse mental health service settings that can inform development of strategic implementation supports. We draw upon the existing literature for three exemplar multilevel service delivery frameworks in different child mental health service settings to (1) identify core components common to each framework, and (2) to highlight prominent implementation determinants that interface with each core component. Six interrelated components of multilevel service delivery frameworks were identified, including, (1) a systems-level approach, (2) data-driven problem solving and decision-making, (3) multiple levels of service intensity using evidence-based practices, (4) cross-linking service sectors, (5) multiple providers working together, including in teams, and (6) built-in implementation strategies that facilitate delivery of the overall model. Implementation determinants that interface with core components were identified at each contextual level. The conceptual foundation provided in this paper has the potential to facilitate cross-sector knowledge sharing, promote generalization across service settings, and provide direction for researchers, system leaders, and implementation intermediaries/practitioners working to strategically support the high-quality implementation of these frameworks.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Human-centered design methods to achieve preparation phase goals in the multiphase optimization strategy framework

Abstract:

Background

The public health impact of behavioral and biobehavioral interventions to prevent and treat mental health and substance use problems hinges on developing methods to strategically maximize their effectiveness, affordability, scalability, and efficiency.

Methods

The multiphase optimization strategy (MOST) is an innovative, principled framework that guides the development of multicomponent interventions. Each phase of MOST (Preparation, Optimization, Evaluation) has explicit goals and a range of appropriate research methods to achieve them. Methods for attaining Optimization and Evaluation phase goals are well-developed. However, methods used in the Preparation phase are often highly researcher-specific, and concrete ways to achieve Preparation phase goals are a priority area for further development.

Results

We propose that the discover, design, build, and test (DDBT) framework provides a theory-driven and methods-rich roadmap for achieving the goals of the Preparation phase of MOST, including specifying the conceptual model, identifying and testing candidate intervention components, and defining the optimization objective. The DDBT framework capitalizes on strategies from the field of human-centered design and implementation science to drive its data collection methods.

Conclusions

MOST and DDBT share many conceptual features, including an explicit focus on implementation determinants, being iterative and flexible, and designing interventions for the greatest public health impact. The proposed synthesized DDBT/MOST approach integrates DDBT into the Preparation phase of MOST thereby providing a framework for rigorous and efficient intervention development research to bolster the success of intervention optimization.

Plain Language Summary

What is already known about the topic? Optimizing behavioral interventions to balance effectiveness with affordability, scalability, and efficiency requires a significant investment in intervention development.

What does this paper add? This paper provides a structured approach to integrating human-centered design principles into the Preparation phase of the multiphase optimization strategy (MOST).

What are the implications for practice, research, or policy? The proposed synthesized model provides a framework for rigorous and efficient intervention development research in the Preparation phase of MOST that will ensure the success of intervention optimization and contribute to improving public health impact of mental health and substance use interventions.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability

Abstract:

Background

Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care.

Method

The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability.

Results

Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes.

Conclusions

CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Harmonizing evidence-based practice, implementation context, and implementation strategies with user-centered design: a case example in young adult cancer care

Abstract:

Background

Attempting to implement evidence-based practices in contexts for which they are not well suited may compromise their fidelity and effectiveness or burden users (e.g., patients, providers, healthcare organizations) with elaborate strategies intended to force implementation. To improve the fit between evidence-based practices and contexts, implementation science experts have called for methods for adapting evidence-based practices and contexts and tailoring implementation strategies; yet, methods for considering the dynamic interplay among evidence-based practices, contexts, and implementation strategies remain lacking. We argue that harmonizing the three can be facilitated by user-centered design, an iterative and highly stakeholder-engaged set of principles and methods.

Methods

This paper presents a case example in which we used a three-phase user-centered design process to design and plan to implement a care coordination intervention for young adults with cancer. Specifically, we used usability testing to redesign and augment an existing patient-reported outcome measure that served as the basis for our intervention to optimize its usability and usefulness, ethnographic contextual inquiry to prepare the context (i.e., a comprehensive cancer center) to promote receptivity to implementation, and iterative prototyping workshops with a multidisciplinary design team to design the care coordination intervention and anticipate implementation strategies needed to enhance contextual fit.

Results

Our user-centered design process resulted in the Young Adult Needs Assessment and Service Bridge (NA-SB), including a patient-reported outcome measure and a collection of referral pathways that are triggered by the needs young adults report, as well as implementation guidance. By ensuring NA-SB directly responded to features of users and context, we designed NA-SB for implementation, potentially minimizing the strategies needed to address misalignment that may have otherwise existed. Furthermore, we designed NA-SB for scale-up; by engaging users from other cancer programs across the country to identify points of contextual variation which would require flexibility in delivery, we created a tool intended to accommodate diverse contexts.

Conclusions

User-centered design can help maximize usability and usefulness when designing evidence-based practices, preparing contexts, and informing implementation strategies—in effect, harmonizing evidence-based practices, contexts, and implementation strategies to promote implementation and effectiveness.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Dissemination Science in School Mental Health: A Framework for Future Research

✪ Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale

Abstract:

Background

Usability—the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction—may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in this study of the IUS.

Method

A survey administered to 136 medical professionals from 11 primary-care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary-care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.

Results

Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.

Conclusion

This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Plain language abstract

The ease with which evidence-based psychosocial interventions (EBPIs) can be readily adopted and used by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability. No quantitative instruments exist to evaluate the usability of complex health interventions, such as the EBPIs that are commonly used to integrate mental and behavioral health services into primary care. This article describes the evaluation of the first quantitative instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

A Theory-Informed Approach to Locally Managed Learning School Systems: Integrating Treatment Integrity and Youth Mental Health Outcome Data to Promote Youth Mental Health

✪ Protocol for a hybrid type 2 cluster randomized trial of trauma-focused cognitive behavioral therapy and a pragmatic individual-level implementation strategy

Abstract:

Background

More than two-thirds of youth experience trauma during childhood, and up to 1 in 5 of these youth develops posttraumatic stress symptoms that significantly impair their functioning. Although trauma-focused cognitive behavior therapy (TF-CBT) has a strong evidence base, it is rarely adopted, delivered with adequate fidelity, or evaluated in the most common setting where youth access mental health services—schools. Given that individual behavior change is ultimately required for successful implementation, even when organizational factors are firmly in place, focusing on individual-level processes represents a potentially parsimonious approach. Beliefs and Attitudes for Successful Implementation in Schools (BASIS) is a pragmatic, motivationally focused multifaceted strategy that augments training and consultation and is designed to target precise mechanisms of behavior change to produce enhanced implementation and youth clinical outcomes. This study protocol describes a hybrid type 2 effectiveness-implementation trial designed to concurrently evaluate the main effects, mediators, and moderators of both the BASIS implementation strategy on implementation outcomes and TF-CBT on youth mental health outcomes.

Methods

Using a cluster randomized controlled design, this trial will assign school-based mental health (SMH) clinicians and schools to one of three study arms: (a) enhanced treatment-as-usual (TAU), (b) attention control plus TF-CBT, or (c) BASIS+TF-CBT. With a proposed sample of 120 SMH clinicians who will each recruit 4–6 youth with a history of trauma (480 children), this project will gather data across 12 different time points to address two project aims. Aim 1 will evaluate, relative to an enhanced TAU condition, the effects of TF-CBT on identified mechanisms of change, youth mental health outcomes, and intervention costs and cost-effectiveness. Aim 2 will compare the effects of BASIS against an attention control plus TF-CBT condition on theoretical mechanisms of clinician behavior change and implementation outcomes, as well as examine costs and cost-effectiveness.

Discussion

This study will generate critical knowledge about the effectiveness and cost-effectiveness of BASIS—a pragmatic, theory-driven, and generalizable implementation strategy designed to enhance motivation—to increase the yield of evidence-based practice training and consultation, as well as the effectiveness of TF-CBT in a novel service setting.

Trial registration

ClinicalTrials.gov registration number NCT04451161. Registered on June 30, 2020.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Introduction

Method

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Authors:

✪ Open Access

Read the full text in the open access journal Administration and Policy in Mental Health

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Methods

Results

Conclusions

Plain Language Summary

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Method

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the hybrid access journal School Mental Health

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusion

Plain language abstract

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal School Mental Health

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Discussion

Trial registration