Implementation climate matters: Evidence from three new studies

Implementation climate is a construct that figures prominently in the inner setting domain of the Consolidated Framework for Implementation Research (CFIR). Three recently published articles provide quantitative evidence that implementation climate matters. Moreover, its contribution to implementation outcomes differs from the contribution of other organizational climate.

What is Implementation Climate?

In 1996, Katherine Klein and Joanne Sorra developed the construct of implementation climate based on an extensive review of the determinants of effective implementation of information technology, noting that organizations use a wide variety of policies and practices to promote innovation use. Examples include training, technical support, incentives, persuasive communication, end-user participation in decision making, workflow changes, workload changes, alterations in staffing levels, alterations in staffing mix, new reporting requirements, new authority relationships, implementation monitoring, and enforcement procedures.

Not only do organizations vary in their use of specific ‘implementation policies and practices,’ but the effectiveness of these policies and practices varies from organization to organization and innovation to innovation. In light of such diversity in organizational practice and variability in effectiveness, Klein and Sorra proposed the construct of implementation climate to shift attention to the collective influence of the multiple policies and practices that organizations employ to promote innovation use.

Implementation climate is a shared perception among intended users of an innovation that innovation use is expected, supported, and rewarded. The stronger the implementation climate, they asserted, the more consistent, high-quality innovation use will be in an organization. Moreover, if implementation climates of equal strength can result from different combinations of implementation policies and practices, as Klein and Sorra claim, then a focus on implementation climate could bring greater clarity to scientific knowledge about the organizational determinants of innovation implementation.

Implementation climate is a central construct in the inner setting domain of the Consolidated Framework for Implementation Research (CFIR). Over the years, implementation scientists have introduced the construct of implementation climate to the field and situated it within a theory of organizational determinants of implementation effectiveness (Weiner et al, 2009), examined the role of implementation climate qualitatively in case studies (, discussed the meaning and measurement of implementation climate (Weiner et al., 2011), evaluated existing measures of implementation climate, and developed new measures with stronger psychometric properties (Jacobs et al., 2011).

Even with all of this work, implementation scientists had not demonstrated quantitatively that implementation climate matters, nor had they empirically disentangled the specific construct of implementation climate from the broader construct of organizational climate, which captures how the work environment is perceived by organizational members. They have now.

New Evidence from Three Studies

In their 2018 publication, Kea Turner and her colleagues tested Klein and Sorra’s organizational theory of innovation implementation effectiveness in a community pharmacy medication management program. Using hurdle regression analysis, they examined whether organizational determinants, such as implementation climate and innovation-values fit, were associated with effective implementation. They defined effective implementation in two ways: implementation versus non-implementation and program reach (i.e., the proportion of the target population that received the intervention.

Turner’s team observed that implementation climate was positively and significantly associated with implementation versus non-implementation and with program reach. In addition, they observed that innovation-values—the extent to which innovation use is consistent with intended users’ values—moderated the relationship between implementation climate and implementation effectiveness, just as the theory predicted.

For implementation climate researchers, these are exciting findings as they add to prior qualitative research indicating that implementation climate matters. Moreover, as Turner and her colleagues note, implementation climate is a modifiable factor that can be targeted through intervention with implementation strategies.

In a second recently published study, Nate Williams and his colleagues tested the hypothesis that organizational climate and implementation climate have joint, cross-level effects on clinicians’ implementation of evidence-based practices in behavioral health organizations. Specifically, they proposed that organizational climate moderates implementation climate’s current and long-term relationships with clinicians’ use of evidence-based practice such that strategic implementation climate will have its most positive effects when it is accompanied by a positive organizational climate.

Sure enough, they observed that in organizations with more positive organizational climates at baseline, higher levels of implementation climate predicted increased evidence-based practice use among clinicians who were present at baseline and among clinicians who were present in the organizations at 2-year follow-up. However, in organizations with less positive organizational climates, implementation climate was not related to clinicians’ use of evidence-based practice at either time point.

Again, these are exciting findings as they demonstrate that organizational climate and implementation climate are distinct constructs. Moreover, optimizing implementation requires attention to both constructs, as strategies that focus solely on strengthening implementation climate might not promote effective implementation unless the organization possesses or engenders a positive organizational climate.

In a third recently published study, Michael Pullman and his colleagues tested the hypothesis that implementation climate is associated with the intensity of workplace-based clinical supervision for evidence-based treatment delivery for children. They noted that workplace-based clinical supervision, in which supervisors provide oversight, feedback, and training to clinicians on clinical practice, is a promising strategy for supporting high-fidelity implementation of evidence-based mental health treatment for children, such as trauma-focused cognitive behavioral therapy (TF-CBT). In a state-funded EBT training initiative in public mental health in Washington State, they examined whether positive implementation climate supported more intense (i.e., frequent and thorough) coverage of core TF-CBT content areas in clinical supervision sessions. A nifty feature of this study is that the authors captured the intensity of supervisors’ coverage of core TF-CBT content areas not with self-reported measures from supervisors but with coded audio-recordings of supervisory sessions with clinicians.

Using three-level mixed effects models, Pullman and his colleagues found that implementation climate was significantly and positively associated two core TF-CBT content areas: exposure, a clinical intervention component and active ingredient of TF-CBT; and assessment, a structural element that supports TF-CBT delivery by guiding treatment decisions and monitoring client progress. They concluded that “a climate that supports, expects, and rewards EBT use may be one of the most important factors for improving the degree to which supervisors cover EBT in their supervision sessions.”

Conclusion

The results of these three studies have significant implications not only for the development of implementation theory, but also the deployment of implementation strategies in organizations seeking to promote the use or delivery of evidence-based practices in health and mental health.

Author: Dr. Bryan J. Weiner