Implementation Climate Archives - Implementation Science at UW

✪ Enduring contributions to implementation science: Honoring the legacy of Bryan R. Garner

With heavy hearts, we are writing to celebrate the scientific contributions of our colleague and friend, Bryan R. Garner, PhD, who passed away incomprehensibly at the age of 49 on June 20, 2025. Bryan was a professor at The Ohio State University, where he served as Director of Dissemination & Implementation Science in the Division of General Internal Medicine and Director of Dissemination & Implementation Research for the Center for the Advancement of Team Science, Analytics, and Systems Thinking (CATALYST). Bryan formed enduring collaborations and friendships across the institutions where he worked and trained. He maintained career-long relationships with colleagues from Texas Christian University, where he earned his PhD in Experimental Psychology (1–3), and from Chestnut Health Systems (4–7) and RTI International (8–10), where he built and expanded his research program.

✪ Study protocol for testing the efficacy of the Helping Educational Leaders Mobilize Evidence (HELM) implementation strategy in elementary schools: a hybrid type 3 effectiveness-implementation randomized controlled trial

Abstract:

Background

Schools need to implement universal student supports that prevent social, emotional, and behavioral difficulties; minimize associated risks; and promote social, emotional, and behavioral competencies. The purpose of this study is to examine the efficacy of the Helping Educational Leaders Mobilize Evidence (HELM) implementation strategy for promoting school-level implementation leadership, implementation climate, and high-fidelity delivery of an evidence-based practice. We will test HELM with an exemplar EBP, Positive Behavioral Interventions and Supports (PBIS). The specific aims of the study are to: 1) experimentally evaluate the effects of HELM versus PBIS training and technical assistance only (control condition); and 2) explore for whom, under what conditions, how equitably, and through which processes HELM works to improve outcomes, as well as its cost-effectiveness.

Methods

This study will use a hybrid type 3 effectiveness-implementation trial to provide a rigorous test of the effects of HELM in elementary schools. Schools will be randomly assigned to HELM + PBIS training and technical assistance (n = 21 schools; n = 210 educators) or PBIS training and technical assistance only (n = 21 schools; n = 210 educators) in a 1:1 ratio within cohorts using covariate constrained randomization that accounts for degree of prior PBIS exposure (measured using the Tiered Fidelity Inventory at baseline) and school size. A series of mixed effects models (time within educator, educator within school) will test within-subject/between-subject interactions across three timepoints (12 months total) to examine whether HELM will show steeper gains than the control on implementation leadership (primary outcome), implementation climate, PBIS fidelity, and student outcomes. Mediational analyses will test hypothesized mechanisms of change (i.e., implementation leadership and climate) of HELM on PBIS fidelity. Sequential mixed-methods data collection and analyses will further explore how organizational mechanisms are linked to implementation outcomes. Cost-effectiveness analyses will compare costs and outcomes of PBIS training and technical assistance only versus PBIS implementation with HELM.

Discussion

The nature of leadership support in schools can make the difference between successful and unsuccessful EBP implementation. Testing HELM within the context of PBIS implementation will provide rigorous evidence about whether and how HELM can equitably address important EBP and student outcomes.

Trial Registration

Clinical Trials ID: NCT06586723. Date of Registration: August 27, 2024. Prospectively registered. URL of Trial Registry Record: https://clinicaltrials.gov/study/NCT06586723?intr=helm&rank=1

Contributions to the Literature

Few effective strategies to enhance evidence-based practice implementation in schools address implementation leadership and climate, and those that do have only been evaluated qualitatively.
Schools have unique organizational characteristics, and it is important to test whether organizational implementation strategies developed in other contexts can be effective in improving implementation outcomes in schools.
This study will evaluate the effects of a tailored adaptation of the Leadership and Organizational Change for Implementation (LOCI) strategy for schools, entitled Helping Educational Leaders Mobilize Evidence (HELM), on implementation mechanisms (implementation leadership and implementation climate), PBIS fidelity, and student outcomes.
This study also will explore for whom, under what conditions, and how equitably HELM demonstrates its effects, providing critical information about influence on – and the limitations of – the impact of this organizational implementation strategy.
Finally, the study will evaluate the processes through which HELM works to improve outcomes, contributing to the emerging literature on implementation mechanisms, as well as its cost-effectiveness.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Name of the registry
clinicaltrials.gov.

Peer Review reports

✪ Not getting better but not getting worse: A cluster randomized controlled pilot trial of a leadership implementation strategy

Abstract:

Background

Implementation of evidence-based practices (EBPs) in schools is fraught with challenges. Even when EBPs are initiated, deterioration of implementation efforts often hinders their long-term success. School leadership behaviors can influence teachers’ EBP implementation. Our study tested an implementation strategy called Helping Educational Leaders Mobilize Evidence (HELM), adapted from the Leadership and Organizational Change for Implementation strategy, to enhance EBP implementation through improvements in school leadership teams’ implementation leadership and climate to buffer against the deterioration of implementation efforts. This study explores the impact of HELM on theorized mechanisms of change (i.e., implementation leadership, climate), educator-level factors (i.e., implementation citizenship), and implementation outcomes (i.e., fidelity, initiative stability).

Method

One school district and 10 schools in Washington participated. Five of the schools were randomized to receive the HELM strategy and the remaining five schools received an alternative leadership training as an implementation attention control. Teachers at every school (n = 341) received training for an EBP called Positive Greetings at the Door that has been previously demonstrated to reduce student behavior problems. Principals and Assistant Principals (n = 18) received the HELM strategy or alternative leadership training. Three district Administrators also participated in HELM as part of the Organizational Strategy Development meetings.

Results

HELM significantly slowed the average decline of implementation leadership (perseverant leadership and communication), three dimensions of implementation climate (recognition, rewards, and existing supports) and total implementation climate, and one dimension of implementation citizenship (keeping informed). No significant effects were found with regard to implementation outcomes (i.e., fidelity, initiative stability).

Conclusions

HELM shows promise in buffering the deterioration of EBP implementation efforts in schools. HELM positively influenced implementation leadership and climate, which are the hypothesized mechanisms for promoting successful long-term implementation efforts. An appropriately powered trial is needed to determine the efficacy of HELM in the future.

Trial Registration Information

Name of the registry: clinicaltrials.gov
Trial registration number: NCT06340074
Date of registration: March 29, 2024. Retrospectively registered
URL of trial registry record: https://clinicaltrials.gov/study/NCT06340074?intr=helm&rank=

Plain Language Summary

Using practices that have been proven to work in research to support youth social, emotional, and behavioral well-being in schools is hard. Efforts to use these practices decrease across the school year. School leadership behaviors can influence educators’ use of these practices. Our study tested an implementation strategy or support designed to help educators use a research-backed practice called Helping Educational Leaders Mobilize Evidence (HELM). HELM was adapted from the Leadership and Organizational Change for Implementation (LOCI) strategy, to enhance school leadership teams’ specific leadership behaviors to protect against the decline in implementation effort. This study explores the impact of HELM on outcomes related to practice use. One school district and 10 schools in western Washington State participated. Five of the schools received the HELM strategy and the remaining five schools received an alternative leadership training. Teachers at every school (n = 341) received training in a practice called Positive Greetings at the Door (PGD) that has been shown to reduce student behavior problems. Principals and Assistant Principals (n = 18) received the HELM strategy or alternative leadership training. Three district administrators also participated in HELM. HELM unexpectedly slowed the average decline of implementation leadership (perseverant leadership and communication), three dimensions of implementation climate (recognition, rewards, and existing supports), total implementation climate, and one dimension of implementation citizenship (keeping informed). No significant effects were found with regard to use of the practice. The HELM strategy shows promise in slowing the deterioration of practice in schools and positively influenced implementation leadership behaviors.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Study protocol for a hybrid type 3 effectiveness-implementation trial of a team-based implementation strategy to support educators’ use of a social engagement intervention

Abstract:

Background

Remaking Recess (RR) is a school-based evidence-based peer social engagement intervention for autistic students. RR involves direct training and coaching with educators; however, educators face several barriers to implementation at both the individual- and organizational-levels. This protocol paper describes a multi-site study that will test whether an educator-level implementation strategy, coaching, with or without a school-level implementation strategy, school-based teams, will maximize educators’ use (fidelity and sustainment) of RR for autistic students and their peers who are socially-isolated, rejected, or peripheral and may need additional support during recess.

Methods

This study will employ a hybrid type-3 effectiveness-implementation trial. Fifty-five elementary schools will be recruited as well as 121 educators (e.g., classroom assistants, aides), 55 general and special educator teachers, and 83–138 other school personnel (e.g., administrators). Additionally, at least 118 autistic students and allistic or non-autistic classmates will be recruited as RR recipients. Participants will complete baseline assessments at the beginning of the year, and all schools will be provided RR training. Schools will be randomized to coaching with or without school-based teams. This study will measure RR fidelity (primary outcome), RR sustainment, as well as peer engagement, social network inclusion, and social skills (secondary outcomes). It is expected that coaching with school-based teams will improve both RR fidelity and social network inclusion, while coaching with and without school-based teams will result in improved peer engagement and social skills.

Discussion

Previous research has documented barriers to RR implementation at both the individual- (provider) and organization-level (school). Using multi-level implementation strategies such as coaching with school-based teams may address these barriers and support RR implementation in schools. Findings from this study may guide future efforts to scale up tailored implementation strategies for use in public school districts, with the ultimate goal of increasing intervention access and improving student outcomes.

Trial registration

Name of the Registry: clinicaltrials.gov.
Trial Registration: Clinical Trials ID: NCT06559267.
Date of Registration: August 15, 2024. Prospectively registered.

Contributions to the literature

Remaking Recess (RR) is an intervention that improves peer engagement and classroom inclusion in autistic students. Educators, who play an important role in supporting autistic youth and their peers who are socially-isolated, rejected, or peripheral during the school day and at recess, can deliver RR.
Schools face many challenges, at both the individual- (i.e., educator) and organizational-levels (i.e., school), when implementing new interventions.
This study will evaluate the impact of two implementation strategies on the fidelity and sustainment of RR: (a) individual direct coaching for educators (coaching), and (b) supporting a team of school personnel (school-based teams) in addition to coaching.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Operational and organizational variation in determinants of policy implementation success: the case of policies that earmark taxes for behavioral health services

Abstract:

Background

Research on determinants of health policy implementation is limited, and conceptualizations of evidence and implementation success are evolving in the field. This study aimed to identify determinants of perceived policy implementation success and assess whether these determinants vary according to: (1) how policy implementation success is operationally defined [i.e., broadly vs. narrowly related to evidence-based practice (EBP) reach] and (2) the role of a person’s organization in policy implementation. The study focuses on policies that earmark taxes for behavioral health services.

Methods

Web-based surveys of professionals involved with earmarked tax policy implementation were conducted between 2022 and 2023 (N = 272). The primary dependent variable was a 9-item score that broadly assessed perceptions of the tax policy positively impacting multiple dimensions of outcomes. The secondary dependent variable was a single item that narrowly assessed perceptions of the tax policy increasing EBP reach. Independent variables were scores mapped to determinants in the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework. Multiple linear regression estimated associations between measures of determinants and policy implementation success.

Results

Perceptions of tax attributes (innovation determinant), tax EBP implementation climate (inner-context determinant), and inter-agency collaboration in tax policy implementation (outer-context and bridging factor determinant) were significantly associated with perceptions of policy implementation success. However, the magnitude of associations varied according to how success was operationalized and by respondent organization type. For example, the magnitude of the association between tax attributes and implementation success was 42% smaller among respondents at direct service organizations than non-direct service organizations when implementation success was operationalized broadly in terms of generating positive impacts (β = 0.37 vs. β = 0.64), and 61% smaller when success was operationalized narrowly in terms of EBP reach (β = 0.23 vs. β = 0.59). Conversely, when success was operationalized narrowly as EBP reach, the magnitude of the association between EBP implementation climate and implementation success was large and significant among respondents at direct service organizations while it was not significant among respondents from non-direct service organizations (β = 0.48 vs. β=-0.06).

Conclusion

Determinants of perceived policy implementation success may vary according to how policy implementation success is defined and the role of a person’s organization in policy implementation. This has implications for implementation science and selecting policy implementation strategies.

Contributions to the literature

The current study contributes to limited knowledge focused on determinants of policy implementation and scholarship about broadening definitions of evidence and implementation success in the field of implementation science.
Attributes of earmarked tax policies, tax policy EBP implementation climate, and frequency of inter-agency collaboration in tax policy implementation are significant determinants of tax policy implementation success and could be targeted by policy implementation strategies.
Determinants of policy implementation success vary according to how policy implementation success is operationally defined (broadly vs. narrowly related to EBP reach) and the role of a person’s organization in policy implementation the process.
Linking policy implementation determinants to the EPIS framework advances understanding of how implementation science frameworks can advance the study of policy implementation.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Relevance of learning health systems to physiatrists and its synergy with implementation science: A commentary

Abstract:

As health care attempts to bridge the gap between evidence and practice, the concept of the learning health system (LHS) is becoming increasingly relevant. LHS integrates evidence with health systems data, driving health care quality and outcomes through updates in policy, practice, and care delivery. In addition, LHS research is becoming critically important as there are several initiatives underway to increase research capacity, expertise, and implementation, including attempts to stimulate increasing numbers of LHS researchers. Physical Medicine & Rehabilitation (PM&R) physicians (physiatrists), nurses, therapists (physical therapists, occupational therapists, speech therapists, clinical psychologists), and scientists are affiliated with LHSs. As LHS research expands in health care systems, better awareness and understanding of LHSs and LHS research competencies are key for rehabilitation professionals including physiatrists. To address this need, the Agency of Healthcare Research and Quality (AHRQ) identified 33 core competencies, grouped into eight domains, for training LHS researchers. The domains are: (1) Systems Science; (2) Research Questions and Standards of Scientific Evidence; (3) Research Methods; (4) Informatics; (5) Ethics of Research and Implementation in Health Systems; (6) Improvement and Implementation Science; (7) Engagement, Leadership, and Research Management; and the recently added (8) Health and Healthcare Equity and Justice. The purpose of this commentary is to define LHS and its relevance to physiatrists, present the role of implementation science (IS) in LHSs and application of IS principles to design LHSs, illustrate current LHS research in rehabilitation, and discuss potential solutions to improve awareness and to stimulate interest in LHS research and IS among physiatrists in LHSs.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Uncovering determinants of perceived feasibility of TF-CBT through coincidence analysis

Abstract:

Introduction

A mental health provider's perception of how well an intervention can be carried out in their context (i.e., feasibility) is an important implementation outcome. This article aims to identify determinants of feasibility of trauma-focused cognitive behavioral therapy (TF-CBT) through a case-based causal approach.

Methods

Data come from an implementation-effectiveness study in which lay counselors (teachers and community health volunteers) implemented a culturally adapted manualized mental health intervention, TF-CBT, delivered to teens who were previously orphaned and were experiencing posttraumatic stress symptoms and prolonged grief in Western Kenya. The intervention team identified combinations of determinants that led to feasibility among teacher- and community health volunteer-counselors through coincidence analysis.

Results

Among teacher-counselors, organizational-level factors (implementation climate, implementation leadership) determined moderate and high levels of feasibility. Among community health volunteer-counselors, a strong relationship between a clinical supervisor and the supervisee was the most influential determinant of feasibility.

Conclusion

Methodology and findings from this article can guide the assessment of determinants of feasibility and the development of implementation strategies for manualized mental health interventions in contexts like Western Kenya.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Project MIMIC (Maximizing Implementation of Motivational Incentives in Clinics): results of a 28-site cluster-randomized type 3 hybrid trial

Abstract:

Background

Contingency management (CM), a behavioral treatment that incentivizes patients for attaining treatment goals, is a highly effective adjunct to medication for opioid use disorder. However, CM is rarely offered in opioid treatment programs in the United States. In a prior pilot trial, the implementation strategy (didactic workshop + feedback + consultation) delivered by the Addiction Technology Transfer Centers (ATTC strategy) promoted CM adoption more effectively than didactic training, but the speed and duration of implementation were sub-optimal. This 28-site type 3 hybrid trial tested the comparative effectiveness of the ATTC strategy versus an Enhanced-ATTC (E-ATTC) strategy that contained two theory-driven techniques targeting implementation climate to improve acceleration and sustainment, respectively: a provider-focused incentivization strategy and a team-focused facilitation strategy. We hypothesized that the E-ATTC strategy would be associated with superior implementation and patient outcomes.

Methods

Twenty-eight opioid treatment programs, 186 providers, and 592 patients were cluster-randomized to receive either the ATTC or E-ATTC strategy. Providers logged their CM sessions in an online CM Tracker and submitted audio-recorded CM sessions, and patients completed surveys about their opioid use at three timepoints. Intention-to-treat analyses examined impacts of the two multi-level strategies on implementation outcomes (CM Exposure, CM Competence, CM Sustainment) and patient outcomes (Opioid Abstinence, Opioid Related Problems).

Results

The pattern of results was identical across unadjusted, propensity score-adjusted, and covariate-adjusted general linear mixed models, though significance varied slightly. Relative to providers receiving the ATTC strategy, those receiving the E-ATTC strategy had significantly higher odds of CM Exposure (covariate adjusted OR = 3.21, p < 0.05) and of attaining the Excellent CM Competence benchmark (propensity-adjusted OR = 4.07, p < 0.05). Patients at the E-ATTC sites had significantly greater likelihood of Opioid Abstinence over time (OR = 2.04, p < 0.05). There were no significant conditional differences in CM Sustainment, though data were measured at the program-level, which limited power to detect differences.

Conclusions

The theory-driven E-ATTC strategy, which targeted implementation climate via facilitation and incentivization, had superior implementation and patient outcomes relative to the ATTC strategy. Results of this study can help inform ongoing CM implementation efforts across the United States.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Frontiers in Health Services

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Discussion

Trial Registration

Contributions to the Literature

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusions

Trial Registration Information

Plain Language Summary

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Discussion

Trial registration

Contributions to the literature

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Results

Conclusion

Contributions to the literature

Authors:

✪ Open Access

Read the full text open access in The Journal of Injury, Function, and Rehabilitation

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Introduction

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Results

Conclusions

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal School Mental Health

Authors:

Read the full text in the subscription access journal Autism