Pragmatic Measurement Archives - Implementation Science at UW

✪ Enduring contributions to implementation science: Honoring the legacy of Bryan R. Garner

With heavy hearts, we are writing to celebrate the scientific contributions of our colleague and friend, Bryan R. Garner, PhD, who passed away incomprehensibly at the age of 49 on June 20, 2025. Bryan was a professor at The Ohio State University, where he served as Director of Dissemination & Implementation Science in the Division of General Internal Medicine and Director of Dissemination & Implementation Research for the Center for the Advancement of Team Science, Analytics, and Systems Thinking (CATALYST). Bryan formed enduring collaborations and friendships across the institutions where he worked and trained. He maintained career-long relationships with colleagues from Texas Christian University, where he earned his PhD in Experimental Psychology (1–3), and from Chestnut Health Systems (4–7) and RTI International (8–10), where he built and expanded his research program.

✪ “Going above and beyond” for implementation in the education sector: extension and validation of the School Implementation Citizenship Behavior Scale (SICBS)

Abstract:

Background

Employee behaviors that strategically support implementation (i.e., implementation citizenship behavior [ICB]) theoretically promote the adoption and high-fidelity use of evidence-based practices (EBPs). ICB (e.g., helping colleagues overcome implementation barriers) may vary across contexts, including schools where children are most likely to access and receive mental and behavioral health services. Pragmatic measures are needed to advance nascent research on school-based ICB and inform how these behaviors can be used to support successful implementation. The current study expanded the Implementation Citizenship Behavior Scale (ICBS) to create and validate the School Implementation Citizenship Behavior Scale (SICBS) in a sample of elementary school teachers implementing evidence-based prevention programs to support children’s mental and behavioral health.

Methods

Based on subject matter expert feedback, items were refined from the original ICBS and items for two new subscales (taking initiative, advocacy) were created for the SICBS. A sample of 441 public school teachers from 52 elementary schools in the Midwest and Western United States of America completed a survey that included the SICBS and additional measures to assess convergent and divergent validity. SICBS was refined and validated via examination of item characteristics curves to reduce items and develop a pragmatic instrument, confirmatory factor analyses to evaluate the hypothesized measurement structure, and assessment of convergent and divergent validity.

Results

The original two ICBS subscales (helping others, keeping informed) were retained, and two new three-item subscales resulted from item reduction analyses (taking initiative, advocacy). The hypothesized second-order factor model was generally well fit to the data (CFI = .99, TLI = .99, RMSEA = .09), all first- (λs = .85-.96) and second-order factor loadings (λs = .93-.95) were high. All SICBS subscales demonstrated acceptable reliability (αs = .88-.92). Convergent validity was evidenced by moderate correlations with organizational citizenship behavior items (rs = .42-.49). Divergent validity was demonstrated by weak correlations with teachers’ beliefs about teaching (rs = .31-.38) and null correlations with most school demographics.

Conclusion

Results support the structural, convergent, and divergent validity of the 12-item, 4-factor SICBS. The SICBS provides a deeper understanding of individual implementer actions that may serve as implementation mechanisms or outcomes.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ A framework for designing hybrid effectiveness-implementation trials for digital health interventions

Abstract:

This article proposes methods for designing randomized controlled trials studying the implementation and effectiveness of digital interventions, meaning websites or applications ("apps") that patients use in healthcare. Deploying digital interventions for behavioral health differs from implementing traditional interventions such as medications or human-delivered therapy. Prior trial design guidance has ignored the existence of international governmental evidence standards, has paid insufficient attention to implementation reporting guidelines, and has not described methods for empirically testing the approach for organizing the delivery of digital interventions.

This framework for designing hybrid effectiveness-implementation trials of digital behavioral health interventions helps researchers articulate research questions that matter to decision-makers and meaningfully contribute to implementation. The framework outlines three phases: 1) frame effectiveness and implementation questions in terms of the digital intervention components, types of clinical support for the digital intervention, and specific strategies for implementing the digital intervention; 2) define and delineate actors, activities, action targets, dose, temporality, and outcomes to maximize inference and reproducibility; and 3) specify trial design features used for hybrid classification.

We illustrate the utility of this framework with two effectiveness-implementation studies of digital interventions for substance use. This framework can help researchers decide on appropriate methodology and help decision-makers apply findings.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Treating seizures faster: The quality improvement in time to treat status epilepticus (QuITT-SE) multicenter randomized stepped wedge clinical trial protocol

✪ Digital communication between mothers and community health workers to support neonatal health (CHV-NEO): study protocol for a randomized controlled trial

Authors:

Keshet Ronen, Lincoln C. Pothan, Violet Apondi, Felix A. Otieno, Daniel Mwakanema, Felix O. Otieno, Lusi Osborn, Julia C. Dettinger, Priyanka Shrestha, Helena Manguerra, Ferdinand Mukumbang, Millicent Masinde, Evelyn Waweru, Mercy Amulele, Christine Were, Beatrice Wasunna, Grace John-Stewart, Bryan Weiner, Arianna Rubin Means, Barbra A. Richardson, Anna B. Hedstrom, Jennifer A. Unger & John Kinuthia

^{University of Washington affiliated authors are displayed in bold.}

✪ Open Access

Published: October 2024

Read the full text in the open access journal Trials

Abstract:

Background

Provision of essential newborn care at home, rapid identification of illness, and care-seeking by caregivers can prevent neonatal mortality. Mobile technology can connect caregivers with information and healthcare worker advice more rapidly and frequently than healthcare visits. Community health workers (CHWs) are well-suited to deliver such interventions. We developed an interactive short message service (SMS) intervention for neonatal health in Kenya, named CHV-NEO. CHV-NEO sends automated, theory-based, actionable, messages throughout the peripartum period that guide mothers to evaluate maternal and neonatal danger signs and facilitate real-time dialogue with a CHW via SMS. We integrated this intervention into Kenya’s national electronic community health information system (eCHIS), which is currently used at scale to support CHW workflow.

Methods

The effect of CHV-NEO on clinical and implementation outcomes will be evaluated through a non-blinded cluster randomized controlled trial. Twenty sites across Kisumu County in Western Kenya were randomized 1:1 to provide either the national eCHIS with integrated CHV-NEO messaging (intervention) or standard of care using eCHIS without CHV-NEO (control). We will compare neonatal mortality between arms based on abstracted eCHIS data from 7200 pregnant women. Secondary outcomes include self-reported provision of essential newborn care (appropriate cord care, thermal care, and timely initiation of breastfeeding), knowledge of neonatal danger signs, and care-seeking for neonatal illness, compared between arms based on questionnaires with a subgroup of 2000 women attending study visits at enrollment in pregnancy and 6 weeks postpartum. We will also determine CHV-NEO’s effect on CHW workflows and evaluate determinants of intervention acceptability, adoption, and fidelity of use through questionnaires, individual interviews, and messaging data.

Discussion

We hypothesize that the CHV-NEO direct-to-client communication strategy can be successfully integrated within existing CHW workflows and infrastructure, improve the provision of at-home essential newborn care, increase timely referral of neonatal illness to facilities, and reduce neonatal mortality. The intervention’s integration into the national eCHIS tool will facilitate rapid scale-up if it is clinically effective and successfully implemented.

Trial registration

ClinicalTrials.gov, NCT05187897. The CHV-NEO study was registered on January 12, 2022.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ A formative mixed methods evaluation of a new Ross program: Why context matters

Abstract:

Background

The Ross procedure is a technically complex operation to address the pathology of the aortic valve. Prior attempts at widespread implementation have been plagued with variations in effectiveness. We report our initial programmatic outcomes using dissemination and implementation (D&I) science frameworks, with an aim to define both programmatic efficacy and effectiveness. The study design was a single center, explanatory sequential mixed methods evaluation of a Ross surgery program from June 2020 to April 2023. Quantitative measures for baseline patient characteristics and postoperative outcomes were summarized. Qualitative measures were obtained using semi-structured interviews and characterized using implementation science frameworks.

Results

A total of 71 patients underwent the Ross procedure at a single academic center. Mean age was 40 years (± 13). There were no perioperative deaths or valve-related reinterventions (0%). During the follow-up period, 2 (2.8%) patients required re-operation and 2 (2.8%) required balloon dilation of the pulmonary homograft. Qualitative measures noted communication and prospective self-audit facilitated measures of effectiveness, as dichotomy within expected vs. observed outcomes led to program modifications. Collaboration across hospital systems promoted adoption and implementation, which led to the incorporation of the Ross procedure into accepted organizational practice.

Conclusions

In this formative evaluation, limited-efficacy outcomes demonstrated similar programmatic success to data from other established centers. With the prerequisite center volume and case selection, the Ross procedure is a safe and efficacious treatment for aortic valvular disease. Implementation science can be used both in real-time and retrospectively to guide organizational efforts to improve outcomes and to improve the translation of results across centers.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Similarities and Differences Between Pragmatic Trials and Hybrid Effectiveness-Implementation Trials

✪ Scaling-up and scaling-out the Systems Analysis and Improvement Approach to optimize the hypertension diagnosis and care cascade for HIV infected individuals (SCALE SAIA-HTN): a stepped-wedge cluster randomized trial

Authors:

Carmen E. Hazim , Igor Dobe, Stephen Pope, Kristjana H. Ásbjörnsdóttir, Orvalho Augusto, Fernando Pereira Bruno, Sergio Chicumbe, Norberto Lumbandali, Inocêncio Mate, Elso Ofumhan, Sam Patel, Riaze Rafik, Kenneth Sherr, Veronica Tonwe, Onei Uetela, David Watkins, Sarah Gimbel, & Ana O. Mocumbi

^{University of Washington affiliated authors are displayed in bold.}

✪ Open Access

Published: March 2024

Read the full text in the open access journal Implementation Science Communication

Abstract:

Background

Undiagnosed and untreated hypertension is a main driver of cardiovascular disease and disproportionately affects persons living with HIV (PLHIV) in low- and middle-income countries. Across sub-Saharan Africa, guideline application to screen and manage hypertension among PLHIV is inconsistent due to poor service readiness, low health worker motivation, and limited integration of hypertension screening and management within HIV care services. In Mozambique, where the adult HIV prevalence is over 13%, an estimated 39% of adults have hypertension. As the only scaled chronic care service in the county, the HIV treatment platform presents an opportunity to standardize and scale hypertension care services.
Low-cost, multi-component systems-level strategies such as the Systems Analysis and Improvement Approach (SAIA) have been found effective at integrating hypertension and HIV services to improve the effectiveness of hypertension care delivery for PLHIV, reduce drop-offs in care, and improve service quality. To build off lessons learned from a recently completed cluster randomized trial (SAIA-HTN) and establish a robust evidence base on the effectiveness of SAIA at scale, we evaluated a scaled-delivery model of SAIA (SCALE SAIA-HTN) using existing district health management structures to facilitate SAIA across six districts of Maputo Province, Mozambique.

Methods

This study employs a stepped-wedge design with randomization at the district level. The SAIA strategy will be “scaled up” with delivery by district health supervisors (rather than research staff) and will be “scaled out” via expansion to Southern Mozambique, to 18 facilities across six districts in Maputo Province. SCALE SAIA-HTN will be introduced over three, 9-month waves of intensive intervention, where technical support will be provided to facilities and district managers by study team members from the Mozambican National Institute of Health. Our evaluation of SCALE SAIA-HTN will be guided by the RE-AIM framework and will seek to estimate the budget impact from the payer’s perspective.

Discussion

SAIA packages user-friendly systems engineering tools to support decision-making by frontline health workers and to identify low-cost, contextually relevant improvement strategies. By integrating SAIA delivery into routine management structures, this pragmatic trial will determine an effective strategy for national scale-up and inform program planning.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Factors Affecting Post-trial Sustainment or De-implementation of Study Interventions: A Narrative Review

✪ A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers

Abstract:

Introduction

Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms.

Method

Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools–Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure.

Results

Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested.

Conclusion

The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Frontiers in Health Services

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text open access in the journal Annals of Epidemiology

Abstract:

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal Contemporary Clinical Trials

Authors:

✪ Open Access

Read the full text in the open access journal Trials

Abstract:

Background

Methods

Discussion

Trial registration

Authors:

✪ Open Access

Read the full text in the open access journal The Cardiothoracic Surgeon

Abstract:

Background

Results

Conclusions

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal Journal of General Internal Medicine

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communication

Abstract:

Background

Methods

Discussion

Authors:

Subscription Required for Full Text Access

Read the full text in the Journal of General Internal Medicine

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Introduction

Method

Results

Conclusion