Michael D. Pullmann Archives - Implementation Science at UW

✪ A randomized controlled trial testing supervision strategies in community mental health

Abstract:

Background

Clinicians need supports beyond training to deliver evidence-based treatments with fidelity. Workplace-based clinical supervision often is a commonly provided support in community mental health, yet too few studies have empirically examined supervision and its impact on clinician fidelity and treatment delivery.

Method

Building on a Washington State-funded evidence-based treatment initiative (CBT+), we conducted a randomized controlled trial (RCT), testing two supervision conditions delivered by workplace-based supervisors (supervisors employed by community mental health organizations). The RCT followed a supervision-as-usual (SAU) phase for comparison. The treatment of focus was trauma-focused cognitive behavioral therapy (TF-CBT). Clinicians (N = 238) from 25 organizations participated in the study across the SAU baseline and RCT phases. In the RCT phase, clinicians were randomized to either symptom and fidelity monitoring (SFM) or SFM and behavioral rehearsal (SFM + BR). For BR, clinicians engaged in a short role play of an upcoming treatment element. Supervisors delivered both conditions, with regular study monitoring for drift. Clinicians audiorecorded therapy sessions with enrolled clients, and masked coders coded a subset of recordings for adherence to TF-CBT. One hundred and thirty-three clinicians had recorded TF-CBT session data for 258 youth. We examined six adherence outcomes, including potential moderators.

Results

Results of generalized estimating equations indicated that there were no real differences on adherence outcomes for experimental conditions (SFM, SFM + BR) compared to SAU. Adherence scores in the baseline SAU phase and the RCT conditions were high. Only one interaction was significant.

Conclusions

Contrary to our hypotheses, we did not see improvements in adherence with the RCT conditions. However, nonsignificant findings seem best explained by clinicians’ acceptable/high adherence in SAU. This study was conducted within the context of a long-standing, state-funded EBT initiative, in which clinicians and their supervisors receive training and support, and in which participating community mental health organizations have adopted and supported TF-CBT.

Trial Registration

ClinicalTrials.gov ID: NCT01800266

Plain Language Summary

Clinicians often receive training in evidence-based treatments (EBTs), but training alone is not enough for clinicians to deliver EBTs well. They need other supports. One of the necessary supports is clinical supervision. In public community mental health organizations, which are often resource-constrained, routine clinical supervision is a commonly available support. However, its potential for supporting high-quality EBT delivery is not well understood. This study looked at supervision and clinicians’ delivery of trauma-focused cognitive behavioral therapy (TF-CBT), an EBT for youth who have mental health problems subsequent to trauma. We found that clinicians who received workplace-based supervision on TF-CBT from TF-CBT-trained supervisors, within the context of a state-funded EBT initiative, delivered TF-CBT well. This initiative supported delivery by providing training and supervisor-specific supports. In a second phase of the study, we rigorously tested supervisors’ use of two specific technique packages. These packages included having supervisors review: (a) graphed client mental health symptoms across treatment and clinicians’ report of TF-CBT elements delivered each session or (b) these plus having clinicians do a brief demonstration of how they would deliver an upcoming TF-CBT element to their client, so supervisors could “see” clinicians’ skill and provide feedback. Contrary to expectations, we did not see important differences in clinicians’ TF-CBT delivery. This may be because the clinicians enrolled in our study were already delivering TF-CBT well and, therefore, there was less opportunity for changes. Future studies should explore if these techniques are more helpful for specific situations or clinicians.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Harnessing Human-Centered Design for Evidence-Based Psychosocial Interventions and Implementation Strategies in Community Settings: Protocol for Redesign to Improve Usability, Engagement, and Appropriateness

Authors:

Aaron R Lyon, Sean A Munson, Michael D Pullmann, Brittany Mosser, Tricia Aung , John Fortney, Alex Dopp, Katie P Osterhage, Helen G Haile, Kathryn E Bruzios, Brittany E Blanchard, Ryan Allred, Macey R Fuller, Patrick J Raue, Ian Bennett, Jill Locke, Karen Bearss, Denise Walker, Elizabeth Connors, Eric J Bruns, Jenna Van Draanen, Doyanne Darnell, Patricia A Areán

^{University of Washington affiliated authors are displayed in bold.}

✪ Open Access

Published: January 2025

Read the full text in the open access journal JMIR Research Protocols

Abstract:

Background

Although substantial progress has been made in establishing evidence-based psychosocial clinical interventions and implementation strategies for mental health, translating research into practice—particularly in more accessible, community settings—has been slow.

Objective

This protocol outlines the renewal of the National Institute of Mental Health–funded University of Washington Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness Center, which draws from human-centered design (HCD) and implementation science to improve clinical interventions and implementation strategies. The Center’s second round of funding (2023-2028) focuses on using the Discover, Design and Build, and Test (DDBT) framework to address 3 priority clinical intervention and implementation strategy mechanisms (ie, usability, engagement, and appropriateness), which we identified as challenges to implementation and scalability during the first iteration of the center. Local redesign teams work collaboratively and share decision-making to carry out DDBT.

Methods

All 4 core studies received institutional review board approval by June 2024, and each pilot project will pursue institutional review board approval when awarded. We will provide research infrastructure to 1 large effectiveness study and 3 exploratory pilot studies as part of the center grant. At least 4 additional small pilot studies will be solicited and funded by the center. All studies will explore the use of DDBT for clinical interventions and implementation strategies to identify modification targets to improve usability, engagement, and appropriateness in accessible nonspecialty settings (Discover phase); develop redesign solutions with local teams to address modification targets (Design and Build phase); and determine if redesign improves usability, engagement, and appropriateness (Test phase), as well as implementation outcomes. Center staff will collaborate with local redesign teams to develop and test clinical interventions and implementation strategies for community settings. We will collaborate with teams to use methods and centerwide measures that facilitate cross-project analysis of the effects of DDBT-driven redesign on outcomes of interest.

Results

As of January 2025, three of the 4 core studies are underway. We will generate additional evidence on the robustness of DDBT and whether combining HCD and implementation science is an asset for improving clinical interventions and implementation strategies.

Conclusions

During the first round of the center, we established that DDBT is a useful approach to systematically identify and address chronic challenges of implementing clinical interventions and implementation strategies. In this subsequent grant, we expect to increase evidence of DDBT’s impact on clinical interventions and implementation strategies by expanding a list of common challenges that could benefit from modification, a list of exemplary solutions to address these challenges, and guidance on using the DDBT framework. These resources will contribute to broader discourse on how to enhance implementation of clinical interventions and implementation strategies that integrate HCD and implementation science.

International Registered Report Identifier (IRRID)

PRR1-10.2196/65446

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): a modified Delphi study

Abstract:

Background

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

Method

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability and inclusion (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

Results

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

Discussion

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

Contributions to the literature

Adaptation of evidence-based practices, programs, and treatments is often needed to ensure successful implementation. However, existing theories/models/frameworks do not outline concrete adaptation steps that are feasible for real-world settings.
We created a resource-efficient method (MODIFI) for developing and assessing intervention adaptations in local settings. This method was iteratively developed through a mixed-methods modified Delphi process, which resulted in consensus among experts in implementation science and practice.
MODIFI leverages human centered design techniques and is tailored to the needs of intervention implementers in applied settings, filling a crucial gap in the research to practice pipeline.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): a modified Delphi study

Abstract:

Background

Method

Results

Discussion

Contributions to the literature

Adaptation of evidence-based practices, programs, and treatments is often needed to ensure successful implementation. However, existing theories/models/frameworks do not outline concrete adaptation steps that are feasible for real-world settings.
We created a resource-efficient method (MODIFI) for developing and assessing intervention adaptations in local settings. This method was iteratively developed through a mixed-methods modified Delphi process, which resulted in consensus among experts in implementation science and practice.
MODIFI leverages human centered design techniques and is tailored to the needs of intervention implementers in applied settings, filling a crucial gap in the research to practice pipeline.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Getting cozy with causality: Advances to the causal pathway diagramming method to enhance implementation precision

Abstract:

Background

Implementation strategies are theorized to work well when carefully matched to implementation determinants and when factors—preconditions, moderators, etc.—that influence strategy effectiveness are prospectively identified and addressed. Existing methods for strategy selection are either imprecise or require significant technical expertise and resources, undermining their utility. This article outlines refinements to causal pathway diagrams (CPDs), a method for articulating the causal process through which implementation strategies work and offers illustrations of their use.

Method

CPDs are a visualization tool to represent an implementation strategy, its mechanism(s) (i.e., the processes through which a strategy is thought to operate), determinants it is intended to address, factors that may impede or facilitate its effectiveness, and the series of outcomes that should be expected if the strategy is operating as intended. We offer principles for constructing CPDs and describe their key functions.

Results

Applications of the CPD method by study teams from two National Institute of Health-funded Implementation Science Centers and a research grant are presented. These include the use of CPDs to (a) match implementation strategies to determinants, (b) understand the conditions under which an implementation strategy works, and (c) develop causal theories of implementation strategies.

Conclusions

CPDs offer a novel method for implementers to select, understand, and improve the effectiveness of implementation strategies. They make explicit theoretical assumptions about strategy operation while supporting practical planning. Early applications have led to method refinements and guidance for the field.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Protocol for a hybrid type 3 effectiveness-implementation trial of a pragmatic individual-level implementation strategy for supporting school-based prevention programming

Abstract:

Background

For approximately one in five children who have social, emotional, and behavioral (SEB) challenges, accessible evidence-based prevention practices (EBPPs) are critical. In the USA, schools are the primary setting for children’s SEB service delivery. Still, EBPPs are rarely adopted and implemented by front-line educators (e.g., teachers) with sufficient fidelity to see effects. Given that individual behavior change is ultimately required for successful implementation, focusing on individual-level processes holds promise as a parsimonious approach to enhance impact.

Beliefs and Attitudes for Successful Implementation in Schools for Teachers (BASIS-T) is a pragmatic, multifaceted pre-implementation strategy targeting volitional and motivational mechanisms of educators’ behavior change to enhance implementation and student SEB outcomes. This study protocol describes a hybrid type 3 effectiveness-implementation trial designed to evaluate the main effects, mediators, and moderators of the BASIS-T implementation strategy as applied to Positive Greetings at the Door, a universal school-based EBPP previously demonstrated to reduce student disruptive behavior and increase academic engagement.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Study protocol: Novel Methods for Implementing Measurement-Based Care with youth in Low-Resource Environments (NIMBLE)

Abstract:

Introduction

For youth receiving care in community mental health centers, comorbidities are the rule rather than the exception. Using measurement-based care (MBC), or the routine evaluation of symptoms to inform care decisions, as the foundation of treatment for youth with comorbid problems significantly improves the impact of psychotherapy by focusing care and building engagement and alliance. MBC increases the rate of symptom improvement, detects clients who would otherwise deteriorate, and alerts clinicians to non-responders. Despite its demonstrated utility, MBC is rarely implemented with fidelity; less than 15% of providers report using MBC per recommendations. Previous efforts to support MBC implementation have yielded suboptimal outcomes, in part, due to organizations’ challenges with identifying and prioritizing barriers and selecting and developing strategies to overcome them. New methods are needed for identifying and prioritizing barriers, and matching strategies to barriers to optimize MBC implementation and treatment quality to improve youth mental health outcomes in community settings.

Methods

Pragmatic implementation methods will be piloted in four diverse community mental health centers. Methods include (a) rapid evidence synthesis; (b) rapid ethnography; (c) design kits (e.g., kits with disposable cameras, journals, maps); (d) barrier prioritization, and (e) causal pathway diagramming. These activities will generate actionable barriers; subsequently, we will use facilitated group processes to prioritize barriers and develop causal pathway diagrams to match strategies to barriers to create implementation plans that optimize MBC fidelity (Aim 1). We will track strategy deployment for 6 months, then compare MBC fidelity for another 6 months post-implementation with data from 2 years of historical controls (Aim 2). Finally, we will co-design a toolkit for design kit methods with youth and the practice and scientific communities (Aim 3).

Discussion

Optimizing MBC implementation in community mental health centers could transform youth mental health care by ensuring the most pressing symptoms are targeted early in treatment. The discussion section highlights expected challenges and limits to using the five methods, including recruitment and engagement given the high pressure on community mental health settings.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale

Abstract:

Background

Usability—the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction—may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in this study of the IUS.

Method

A survey administered to 136 medical professionals from 11 primary-care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary-care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.

Results

Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.

Conclusion

This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Plain language abstract

The ease with which evidence-based psychosocial interventions (EBPIs) can be readily adopted and used by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability. No quantitative instruments exist to evaluate the usability of complex health interventions, such as the EBPIs that are commonly used to integrate mental and behavioral health services into primary care. This article describes the evaluation of the first quantitative instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Using implementation science to close the gap between the optimal and typical practice of quantitative methods in clinical science

✪ The role of the outer setting in implementation: associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare

Abstract:

Background

Despite consistent recognition of their influence, empirical study of how outer setting factors (e.g., policies, financing, stakeholder relationships) influence public systems’ investment in and adoption of evidence-based treatment (EBT) is limited. This study examined associations among unmodifiable (e.g., demographic, economic, political, structural factors) and modifiable (e.g., allocation of resources, social processes, policies, and regulations) outer setting factors and adoption of behavioral health EBT by US states.

Methods

Multilevel models examined relationships between state characteristics, an array of funding and policy variables, and state adoption of behavioral health EBTs for adults and children across years 2002–2012, using data from the National Association for State Mental Health Program Directors Research Institute and other sources.

Results

Several unmodifiable state factors, including per capita income, controlling political party, and Medicaid expansion, predicted level of state fiscal investments in EBT. By contrast, modifiable factors, such as interagency collaboration and investment in research centers, were more predictive of state policies supportive of EBT. Interestingly, level of adult EBT adoption was associated with state fiscal supports for EBT, while child EBT adoption was predicted more by supportive policies. State per capita debt and direct state operation of services (versus contracting for services) predicted both child and adult EBT adoption.

Conclusions

State-level EBT adoption and associated implementation support is associated with an interpretable array of policy, financing, and oversight factors. Such information expands our knowledge base of the role of the outer setting in implementation and may provide insight into how best to focus efforts to promote EBT for behavioral health disorders.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusions

Trial Registration

Plain Language Summary

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Research Protocols

Abstract:

Background

Objective

Methods

Results

Conclusions

International Registered Report Identifier (IRRID)

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Method

Results

Discussion

Contributions to the literature

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Method

Results

Discussion

Contributions to the literature

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Introduction

Methods

Discussion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusion

Plain language abstract

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access Journal of Abnormal Psychology

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Results

Conclusions