Aaron R. Lyon Archives - Implementation Science at UW

✪ Rigorous projects and grants integrating human-centered design and implementation science: 8 tips for success

Abstract:

Background

Researchers and funders increasingly see potential for integrating human-centered design (HCD) and implementation science (IS) approaches to enhance the design and uptake of health interventions and implementation strategies. These approaches offer complementary strengths. HCD prioritizes developing compelling, intuitive, and adoptable innovations grounded in user and contextual needs; IS focuses on ensuring evidence-based practices reach and work for their intended populations. Integrating HCD and IS creates challenges that existing IS proposal writing guidelines do not address. HCD’s inherent uncertainty–since final designs are developed through iteration rather than pre-determined–can unsettle researchers and funders accustomed to more defined endpoints. Variation in how HCD is understood adds further difficulty when collaborating across disciplines or communicating with funders. There are currently no published recommendations to help researchers design strong studies integrating HCD and IS and communicate them effectively to funders.

Discussion

We present tips to support both rigorous study design and effective proposal writing that integrate HCD and IS. This draws from our experiences leading, supporting, and reviewing both funded and unfunded projects. Our considerations are informed primarily by experiences with teams in the United States in the mental health and behavioral health domains; however, we have found our findings also apply to projects globally and in other health sectors. Our intent is that this guide will advance good research that leverages both approaches and is viable for funding. We also hope that the tips will support funders in developing solicitations and evaluating proposals.

Conclusion

There is untapped potential for innovative health research integrating HCD and IS, but realizing it requires both rigorous study design and an ability to attract funding. We offer practice guidance to help researchers and funders develop quality projects and proposals, and we encourage others to build on these recommendations.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ “Going above and beyond” for implementation in the education sector: extension and validation of the School Implementation Citizenship Behavior Scale (SICBS)

Abstract:

Background

Employee behaviors that strategically support implementation (i.e., implementation citizenship behavior [ICB]) theoretically promote the adoption and high-fidelity use of evidence-based practices (EBPs). ICB (e.g., helping colleagues overcome implementation barriers) may vary across contexts, including schools where children are most likely to access and receive mental and behavioral health services. Pragmatic measures are needed to advance nascent research on school-based ICB and inform how these behaviors can be used to support successful implementation. The current study expanded the Implementation Citizenship Behavior Scale (ICBS) to create and validate the School Implementation Citizenship Behavior Scale (SICBS) in a sample of elementary school teachers implementing evidence-based prevention programs to support children’s mental and behavioral health.

Methods

Based on subject matter expert feedback, items were refined from the original ICBS and items for two new subscales (taking initiative, advocacy) were created for the SICBS. A sample of 441 public school teachers from 52 elementary schools in the Midwest and Western United States of America completed a survey that included the SICBS and additional measures to assess convergent and divergent validity. SICBS was refined and validated via examination of item characteristics curves to reduce items and develop a pragmatic instrument, confirmatory factor analyses to evaluate the hypothesized measurement structure, and assessment of convergent and divergent validity.

Results

The original two ICBS subscales (helping others, keeping informed) were retained, and two new three-item subscales resulted from item reduction analyses (taking initiative, advocacy). The hypothesized second-order factor model was generally well fit to the data (CFI = .99, TLI = .99, RMSEA = .09), all first- (λs = .85-.96) and second-order factor loadings (λs = .93-.95) were high. All SICBS subscales demonstrated acceptable reliability (αs = .88-.92). Convergent validity was evidenced by moderate correlations with organizational citizenship behavior items (rs = .42-.49). Divergent validity was demonstrated by weak correlations with teachers’ beliefs about teaching (rs = .31-.38) and null correlations with most school demographics.

Conclusion

Results support the structural, convergent, and divergent validity of the 12-item, 4-factor SICBS. The SICBS provides a deeper understanding of individual implementer actions that may serve as implementation mechanisms or outcomes.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Study protocol for testing the efficacy of the Helping Educational Leaders Mobilize Evidence (HELM) implementation strategy in elementary schools: a hybrid type 3 effectiveness-implementation randomized controlled trial

Abstract:

Background

Schools need to implement universal student supports that prevent social, emotional, and behavioral difficulties; minimize associated risks; and promote social, emotional, and behavioral competencies. The purpose of this study is to examine the efficacy of the Helping Educational Leaders Mobilize Evidence (HELM) implementation strategy for promoting school-level implementation leadership, implementation climate, and high-fidelity delivery of an evidence-based practice. We will test HELM with an exemplar EBP, Positive Behavioral Interventions and Supports (PBIS). The specific aims of the study are to: 1) experimentally evaluate the effects of HELM versus PBIS training and technical assistance only (control condition); and 2) explore for whom, under what conditions, how equitably, and through which processes HELM works to improve outcomes, as well as its cost-effectiveness.

Methods

This study will use a hybrid type 3 effectiveness-implementation trial to provide a rigorous test of the effects of HELM in elementary schools. Schools will be randomly assigned to HELM + PBIS training and technical assistance (n = 21 schools; n = 210 educators) or PBIS training and technical assistance only (n = 21 schools; n = 210 educators) in a 1:1 ratio within cohorts using covariate constrained randomization that accounts for degree of prior PBIS exposure (measured using the Tiered Fidelity Inventory at baseline) and school size. A series of mixed effects models (time within educator, educator within school) will test within-subject/between-subject interactions across three timepoints (12 months total) to examine whether HELM will show steeper gains than the control on implementation leadership (primary outcome), implementation climate, PBIS fidelity, and student outcomes. Mediational analyses will test hypothesized mechanisms of change (i.e., implementation leadership and climate) of HELM on PBIS fidelity. Sequential mixed-methods data collection and analyses will further explore how organizational mechanisms are linked to implementation outcomes. Cost-effectiveness analyses will compare costs and outcomes of PBIS training and technical assistance only versus PBIS implementation with HELM.

Discussion

The nature of leadership support in schools can make the difference between successful and unsuccessful EBP implementation. Testing HELM within the context of PBIS implementation will provide rigorous evidence about whether and how HELM can equitably address important EBP and student outcomes.

Trial Registration

Clinical Trials ID: NCT06586723. Date of Registration: August 27, 2024. Prospectively registered. URL of Trial Registry Record: https://clinicaltrials.gov/study/NCT06586723?intr=helm&rank=1

Contributions to the Literature

Few effective strategies to enhance evidence-based practice implementation in schools address implementation leadership and climate, and those that do have only been evaluated qualitatively.
Schools have unique organizational characteristics, and it is important to test whether organizational implementation strategies developed in other contexts can be effective in improving implementation outcomes in schools.
This study will evaluate the effects of a tailored adaptation of the Leadership and Organizational Change for Implementation (LOCI) strategy for schools, entitled Helping Educational Leaders Mobilize Evidence (HELM), on implementation mechanisms (implementation leadership and implementation climate), PBIS fidelity, and student outcomes.
This study also will explore for whom, under what conditions, and how equitably HELM demonstrates its effects, providing critical information about influence on – and the limitations of – the impact of this organizational implementation strategy.
Finally, the study will evaluate the processes through which HELM works to improve outcomes, contributing to the emerging literature on implementation mechanisms, as well as its cost-effectiveness.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Name of the registry
clinicaltrials.gov.

Peer Review reports

✪ School-Partnered Collaborative Care (SPACE) for Pediatric Type 1 Diabetes: Development and Usability Study of a Virtual Intervention With Multisystem Community Partners

Abstract:

Background

School-partnered interventions may improve health outcomes for children with type 1 diabetes, though there is limited evidence to support their effectiveness and sustainability. Family, school, or health system factors may interfere with intervention usability and implementation.

Objective

To identify and address potential implementation barriers during intervention development, we combined methods in user-centered design and implementation science to adapt an evidence-based psychosocial intervention, the collaborative care model, to a virtual school-partnered collaborative care (SPACE) model for type 1 diabetes between schools and diabetes medical teams.

Methods

We recruited patient, family, school, and health system partners (n=20) to cocreate SPACE through iterative, web-based design sessions using a digital whiteboard (phase 1). User-centered design methods included independent and group activities for idea generation, visual voting, and structured critique of the evolving SPACE prototype. In phase 2, the prototype was evaluated with the usability evaluation for evidence-based psychosocial interventions methods. School nurses reviewed the prototype and tasks in cognitive walkthroughs and completed the Intervention Usability Scale (IUS). Two members of the research team independently identified and prioritized (1-3 rating) discrete usability concerns. We evaluated the relationship between prioritization and the percentage of nurses reporting each usability issue with Spearman correlation. Differences in IUS scores by school nurse characteristics were assessed with ANOVA.

Results

In the design phase, the partners generated over 90 unique ideas for SPACE, prioritizing elements pertaining to intervention adaptability, team-based communication, and multidimensional outcome tracking. Following three iterations of prototype development, cognitive walkthroughs were completed with 10 school nurses (n=10, 100% female; mean age 48.5, SD 9.5 years) representing different districts and years of experience. Nurses identified 16 discrete usability issues (each reported by 10%-60% of participants). Two issues receiving the highest priority (3.0): ability to access a virtual platform (n=3, 30% of participants) and data-sharing mechanisms between nurses and providers (n=6, 60% of participants). There was a moderate correlation between priority rating and the percentage of nurses reporting each issue (ρ=0.63; P=.01). Average IUS ratings (77.8, SD 11.1; 100-point scale) indicated appropriate usability. There was no difference in IUS ratings by school nurse experience (P=.54), student caseload (P=.12), number of schools covered (P=.90), or prior experience with type 1 diabetes (P=.83), suggesting that other factors may influence usability. The design team recommended strategies for SPACE implementation to overcome high-priority issues, including training users on videoconferencing applications, establishing secure forms for school data reporting, and sharing glucose data in real-time during SPACE meetings.

Conclusions

Cross-sector interventions are complex, and perceived usability is a potential barrier to implementation. Using web-based cocreation methods with community partners promoted high-quality intervention design that is aligned with end-user priorities. Quantitative and qualitative assessments indicated appropriate degree of usability to move forward with pilot-testing.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ A framework for designing hybrid effectiveness-implementation trials for digital health interventions

Abstract:

This article proposes methods for designing randomized controlled trials studying the implementation and effectiveness of digital interventions, meaning websites or applications ("apps") that patients use in healthcare. Deploying digital interventions for behavioral health differs from implementing traditional interventions such as medications or human-delivered therapy. Prior trial design guidance has ignored the existence of international governmental evidence standards, has paid insufficient attention to implementation reporting guidelines, and has not described methods for empirically testing the approach for organizing the delivery of digital interventions.

This framework for designing hybrid effectiveness-implementation trials of digital behavioral health interventions helps researchers articulate research questions that matter to decision-makers and meaningfully contribute to implementation. The framework outlines three phases: 1) frame effectiveness and implementation questions in terms of the digital intervention components, types of clinical support for the digital intervention, and specific strategies for implementing the digital intervention; 2) define and delineate actors, activities, action targets, dose, temporality, and outcomes to maximize inference and reproducibility; and 3) specify trial design features used for hybrid classification.

We illustrate the utility of this framework with two effectiveness-implementation studies of digital interventions for substance use. This framework can help researchers decide on appropriate methodology and help decision-makers apply findings.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Outcome preferences in fidelity-adaptation scenarios across evidence-based parenting programs: A discrete choice experiment

Abstract:

Background

Implementing evidence-based parenting programs often involves navigating fidelity-adaptation decisions. While research has explored various aspects of this dilemma, little is known about how practitioners’ outcome preferences influence their decisions in real-world scenarios.

Methods

This study employed a discrete choice experiment (DCE) to investigate the relative importance of five outcomes (Relationship Quality, Satisfaction, Workload Strain, Value Conflict, and Reach) in fidelity-adaptation decisions among 209 practitioners delivering evidence-based parenting programs in Sweden. The DCE presented 25 choice sets across five contextual scenarios, analyzed using Bayesian hierarchical logistic regression.

Results

All five outcomes significantly influenced practitioners’ choices, with Relationship Quality emerging as the most impactful (log-odds: 4.56, 95% CI [4.16, 4.91]). Satisfaction and minimizing Value Conflict showed similar importance (log odds: 2.45 and -2.40, respectively), while Workload Strain and Reach had slightly less impact (log odds: -2.10 and 1.96, respectively).

Conclusions

This study offers a novel perspective on the role of outcome preference in navigating fidelity-adaptation decisions. The strong preference for improving parent-child relationships aligns with core parenting program goals, while consideration of other outcomes reflects practitioners’ holistic approach to implementation. These findings can inform the design of interventions and implementation strategies that balance effectiveness with real-world constraints, potentially enhancing parenting programs’ adoption, sustainability, and impact.

Contributions to the literature

This study pioneers the application of discrete choice experiments in exploring fidelity-adaptation decisions.
Quantifying practitioners’ outcome preferences provides empirical evidence of complex trade-offs in implementing evidence-based parenting programs.
The study’s results provide actionable insights for program developers, policymakers, and implementation scientists on designing interventions that resonate with practitioners’ priorities while maintaining program effectiveness.
This research bridges theoretical frameworks and real-world decision-making, advancing our understanding of how various factors influence intervention delivery in practice.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Not getting better but not getting worse: A cluster randomized controlled pilot trial of a leadership implementation strategy

Abstract:

Background

Implementation of evidence-based practices (EBPs) in schools is fraught with challenges. Even when EBPs are initiated, deterioration of implementation efforts often hinders their long-term success. School leadership behaviors can influence teachers’ EBP implementation. Our study tested an implementation strategy called Helping Educational Leaders Mobilize Evidence (HELM), adapted from the Leadership and Organizational Change for Implementation strategy, to enhance EBP implementation through improvements in school leadership teams’ implementation leadership and climate to buffer against the deterioration of implementation efforts. This study explores the impact of HELM on theorized mechanisms of change (i.e., implementation leadership, climate), educator-level factors (i.e., implementation citizenship), and implementation outcomes (i.e., fidelity, initiative stability).

Method

One school district and 10 schools in Washington participated. Five of the schools were randomized to receive the HELM strategy and the remaining five schools received an alternative leadership training as an implementation attention control. Teachers at every school (n = 341) received training for an EBP called Positive Greetings at the Door that has been previously demonstrated to reduce student behavior problems. Principals and Assistant Principals (n = 18) received the HELM strategy or alternative leadership training. Three district Administrators also participated in HELM as part of the Organizational Strategy Development meetings.

Results

HELM significantly slowed the average decline of implementation leadership (perseverant leadership and communication), three dimensions of implementation climate (recognition, rewards, and existing supports) and total implementation climate, and one dimension of implementation citizenship (keeping informed). No significant effects were found with regard to implementation outcomes (i.e., fidelity, initiative stability).

Conclusions

HELM shows promise in buffering the deterioration of EBP implementation efforts in schools. HELM positively influenced implementation leadership and climate, which are the hypothesized mechanisms for promoting successful long-term implementation efforts. An appropriately powered trial is needed to determine the efficacy of HELM in the future.

Trial Registration Information

Name of the registry: clinicaltrials.gov
Trial registration number: NCT06340074
Date of registration: March 29, 2024. Retrospectively registered
URL of trial registry record: https://clinicaltrials.gov/study/NCT06340074?intr=helm&rank=

Plain Language Summary

Using practices that have been proven to work in research to support youth social, emotional, and behavioral well-being in schools is hard. Efforts to use these practices decrease across the school year. School leadership behaviors can influence educators’ use of these practices. Our study tested an implementation strategy or support designed to help educators use a research-backed practice called Helping Educational Leaders Mobilize Evidence (HELM). HELM was adapted from the Leadership and Organizational Change for Implementation (LOCI) strategy, to enhance school leadership teams’ specific leadership behaviors to protect against the decline in implementation effort. This study explores the impact of HELM on outcomes related to practice use. One school district and 10 schools in western Washington State participated. Five of the schools received the HELM strategy and the remaining five schools received an alternative leadership training. Teachers at every school (n = 341) received training in a practice called Positive Greetings at the Door (PGD) that has been shown to reduce student behavior problems. Principals and Assistant Principals (n = 18) received the HELM strategy or alternative leadership training. Three district administrators also participated in HELM. HELM unexpectedly slowed the average decline of implementation leadership (perseverant leadership and communication), three dimensions of implementation climate (recognition, rewards, and existing supports), total implementation climate, and one dimension of implementation citizenship (keeping informed). No significant effects were found with regard to use of the practice. The HELM strategy shows promise in slowing the deterioration of practice in schools and positively influenced implementation leadership behaviors.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Harnessing Human-Centered Design for Evidence-Based Psychosocial Interventions and Implementation Strategies in Community Settings: Protocol for Redesign to Improve Usability, Engagement, and Appropriateness

Authors:

Aaron R Lyon, Sean A Munson, Michael D Pullmann, Brittany Mosser, Tricia Aung , John Fortney, Alex Dopp, Katie P Osterhage, Helen G Haile, Kathryn E Bruzios, Brittany E Blanchard, Ryan Allred, Macey R Fuller, Patrick J Raue, Ian Bennett, Jill Locke, Karen Bearss, Denise Walker, Elizabeth Connors, Eric J Bruns, Jenna Van Draanen, Doyanne Darnell, Patricia A Areán

^{University of Washington affiliated authors are displayed in bold.}

✪ Open Access

Published: January 2025

Read the full text in the open access journal JMIR Research Protocols

Abstract:

Background

Although substantial progress has been made in establishing evidence-based psychosocial clinical interventions and implementation strategies for mental health, translating research into practice—particularly in more accessible, community settings—has been slow.

Objective

This protocol outlines the renewal of the National Institute of Mental Health–funded University of Washington Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness Center, which draws from human-centered design (HCD) and implementation science to improve clinical interventions and implementation strategies. The Center’s second round of funding (2023-2028) focuses on using the Discover, Design and Build, and Test (DDBT) framework to address 3 priority clinical intervention and implementation strategy mechanisms (ie, usability, engagement, and appropriateness), which we identified as challenges to implementation and scalability during the first iteration of the center. Local redesign teams work collaboratively and share decision-making to carry out DDBT.

Methods

All 4 core studies received institutional review board approval by June 2024, and each pilot project will pursue institutional review board approval when awarded. We will provide research infrastructure to 1 large effectiveness study and 3 exploratory pilot studies as part of the center grant. At least 4 additional small pilot studies will be solicited and funded by the center. All studies will explore the use of DDBT for clinical interventions and implementation strategies to identify modification targets to improve usability, engagement, and appropriateness in accessible nonspecialty settings (Discover phase); develop redesign solutions with local teams to address modification targets (Design and Build phase); and determine if redesign improves usability, engagement, and appropriateness (Test phase), as well as implementation outcomes. Center staff will collaborate with local redesign teams to develop and test clinical interventions and implementation strategies for community settings. We will collaborate with teams to use methods and centerwide measures that facilitate cross-project analysis of the effects of DDBT-driven redesign on outcomes of interest.

Results

As of January 2025, three of the 4 core studies are underway. We will generate additional evidence on the robustness of DDBT and whether combining HCD and implementation science is an asset for improving clinical interventions and implementation strategies.

Conclusions

During the first round of the center, we established that DDBT is a useful approach to systematically identify and address chronic challenges of implementing clinical interventions and implementation strategies. In this subsequent grant, we expect to increase evidence of DDBT’s impact on clinical interventions and implementation strategies by expanding a list of common challenges that could benefit from modification, a list of exemplary solutions to address these challenges, and guidance on using the DDBT framework. These resources will contribute to broader discourse on how to enhance implementation of clinical interventions and implementation strategies that integrate HCD and implementation science.

International Registered Report Identifier (IRRID)

PRR1-10.2196/65446

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Human-Centered Design to Enhance Implementation and Impact in Health

✪ A research agenda to advance the study of implementation mechanisms

Abstract:

Background

Implementation science scholars have made significant progress identifying factors that enable or obstruct the implementation of evidence-based interventions, and testing strategies that may modify those factors. However, little research sheds light on how or why strategies work, in what contexts, and for whom. Studying implementation mechanisms—the processes responsible for change—is crucial for advancing the field of implementation science and enhancing its value in facilitating equitable policy and practice change. The Agency for Healthcare Research and Quality funded a conference series to achieve two aims: (1) develop a research agenda on implementation mechanisms, and (2) actively disseminate the research agenda to research, policy, and practice audiences. This article presents the resulting research agenda, including priorities and actions to encourage its execution.

Method

Building on prior concept mapping work, in a semi-structured, 3-day, in-person working meeting, 23 US-based researchers used a modified nominal group process to generate priorities and actions for addressing challenges to studying implementation mechanisms. During each of the three 120-min sessions, small groups responded to the prompt: “What actions need to be taken to move this research forward?” The groups brainstormed actions, which were then shared with the full group and discussed with the support of facilitators trained in structured group processes. Facilitators grouped critical and novel ideas into themes. Attendees voted on six themes they prioritized to discuss in a fourth, 120-min session, during which small groups operationalized prioritized actions. Subsequently, all ideas were collated, combined, and revised for clarity by a subset of the authorship team.

Results

From this multistep process, 150 actions emerged across 10 priority areas, which together constitute the research agenda. Actions included discrete activities, projects, or products, and ways to shift how research is conducted to strengthen the study of implementation mechanisms.

Conclusions

This research agenda elevates actions to guide the selection, design, and evaluation of implementation mechanisms. By delineating recommended actions to address the challenges of studying implementation mechanisms, this research agenda facilitates expanding the field of implementation science, beyond studying what works to how and why strategies work, in what contexts, for whom, and with which interventions.

Contributions to the literature

This research agenda operationalizes a set of activities to strengthen the implementation science field’s focus on why and how strategies work.
The research agenda addresses the following activities: accumulating knowledge, innovating methods and overcoming design challenges, improving measurement, providing guidance for specifying causal mechanisms, increasing focus on theorizing, engaging the policy and practice community, engaging funders, building capacity, enhancing equity, and effectively disseminating methods.
Studying implementation mechanisms can promote pragmatic strategy development, equitable processes and outcomes, and policy relevance by clarifying pathways for overcoming contextually specific barriers and achieving outcomes of interest.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Discussion

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Discussion

Trial Registration

Contributions to the Literature

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Diabetes

Abstract:

Background

Objective

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text open access in the journal Annals of Epidemiology

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Results

Conclusions

Contributions to the literature

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusions

Trial Registration Information

Plain Language Summary

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Research Protocols

Abstract:

Background

Objective

Methods

Results

Conclusions

International Registered Report Identifier (IRRID)

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal Annual Review of Public Health

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Method

Results

Conclusions

Contributions to the literature