Skip to content
Implementation Science at UW

Keywords: 2020

✪ Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties

Authors:

Kayne Mettert, Cara C. Lewis, Caitlin Dorsey, Heather Halko, & Bryan J. Weiner

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: August 2020

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Systematic reviews of measures can facilitate advances in implementation research and practice by locating reliable and valid measures and highlighting measurement gaps. Our team completed a systematic review of implementation outcome measures published in 2015 that indicated a severe measurement gap in the field. Now, we offer an update with this enhanced systematic review to identify and evaluate the psychometric properties of measures of eight implementation outcomes used in behavioral health care.

Methods

The systematic review methodology is described in detail in a previously published protocol paper and summarized here. The review proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and measure forward searches. Phase II, data extraction, involved coding psychometric information. Phase III, data analysis, involved two trained specialists independently rating each measure using PAPERS (Psychometric And Pragmatic Evidence Rating Scales).

Results

Searches identified 150 outcomes measures of which 48 were deemed unsuitable for rating and thus excluded, leaving 102 measures for review. We identified measures of acceptability (N = 32), adoption (N = 26), appropriateness (N = 6), cost (N = 31), feasibility (N = 18), fidelity (N = 18), penetration (N = 23), and sustainability (N = 14). Information about internal consistency and norms were available for most measures (59%). Information about other psychometric properties was often not available. Ratings for internal consistency and norms ranged from “adequate” to “excellent.” Ratings for other psychometric properties ranged mostly from “poor” to “good.”

Conclusion

While measures of implementation outcomes used in behavioral health care (including mental health, substance use, and other addictive behaviors) are unevenly distributed and exhibit mostly unknown psychometric quality, the data reported in this article show an overall improvement in availability of psychometric information. This review identified a few promising measures, but targeted efforts are needed to systematically develop and test measures that are useful for both research and practice.

Plain language abstract

When implementing an evidence-based treatment into practice, it is important to assess several outcomes to gauge how effectively it is being implemented. Outcomes such as acceptability, feasibility, and appropriateness may offer insight into why providers do not adopt a new treatment. Similarly, outcomes such as fidelity and penetration may provide important context for why a new treatment did not achieve desired effects. It is important that methods to measure these outcomes are accurate and consistent. Without accurate and consistent measurement, high-quality evaluations cannot be conducted. This systematic review of published studies sought to identify questionnaires (referred to as measures) that ask staff at various levels (e.g., providers, supervisors) questions related to implementation outcomes, and to evaluate the quality of these measures. We identified 150 measures and rated the quality of their evidence with the goal of recommending the best measures for future use. Our findings suggest that a great deal of work is needed to generate evidence for existing measures or build new measures to achieve confidence in our implementation evaluations.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Defining gaps in pre-exposure prophylaxis delivery for pregnant and post-partum women in high-burden settings using an implementation science framework

Authors:

Jillian Pintye, Dvora L. Joseph Davey, Anjuli D. Wagner, Grace John-Stewart, Rachel Baggaley, Linda-Gail Bekker, Connie Celum, Benjamin H. Chi, Thomas J. Coates, Allison K. Groves, Jessica E. Haberer, Renee Heffron, John Kinuthia, Lynn T. Matthews, James A. McIntyre, Dhayendre Moodley, Lynne M. Mofenson, Nelly Mugo, Andrew Mujugira, Landon Myer, Steven Shoptaw, Lynda Stranix-Chibanda, Jared M. Baeten, for the PrEP in Pregnancy Working Group

University of Washington affiliated authors are displayed in bold.

Subscription Required for Full Text Access

Published: August 2020

Read the full text open access in the subscription access journal The Lancet: HIV

✪ Practice facilitation to implement alcohol-related care in Veterans Health Administration liver clinics: a study protocol

Authors:

Madeline C. Frost, George N. Ioannou, Judith I. Tsui, E. Jennifer Edelman, Bryan J. Weiner, Olivia V. Fletcher & Emily C. Williams

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: July 2020

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Alcohol-related care, including screening, brief intervention, and provision of/referral to medication or behavioral treatments for alcohol use disorder, could be delivered in liver clinics to better reach patients with chronic liver conditions. However, the provision of alcohol-related care in liver clinics is currently suboptimal. Practice facilitation is an evidence-based implementation strategy that may address barriers, harness facilitators, and optimize the implementation of alcohol-related care in liver clinic settings using a clinic-centered approach. We report the protocol of a study to test a practice facilitation intervention to implement alcohol-related care in four Veterans Health Administration liver clinics.

Methods

This study will employ a Hybrid Type 3 effectiveness-implementation design, in which implementation outcomes are considered primary and clinical outcomes secondary. Intervention and evaluation design were informed by the Consolidated Framework for Implementation Research. Qualitative data collected from clinical stakeholders and patients were used to tailor the intervention. The intervention involves a 6-month period of external practice facilitation, including regular meetings to identify clinic goals, challenges, and solutions; engagement of clinic champions; provision of training and development of educational materials for clinic staff and patients; and performance monitoring and feedback. Ongoing formative evaluation involves the collection of quantitative facilitator tracking data and qualitative data from meeting notes and patient interviews to describe intervention acceptability, feasibility, and adoption, and adjust implementation as needed. In the summative evaluation, implementation outcomes (clinic rates of screening, brief intervention, and treatment referral/receipt) and clinical outcomes (unhealthy alcohol use, liver health) will be assessed among patients in participating clinics using secondary electronic health record data and interrupted time series analysis.

Discussion

This will be the first study to our knowledge to test practice facilitation to implement alcohol-related care in liver clinic settings. Results from formative and summative evaluation will inform a framework for the successful implementation of effective alcohol-related care through practice facilitation in liver clinics, which may ultimately lead to better health outcomes for patients with chronic liver disease.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

What Gets Measured Gets Done: How Mental Health Agencies can Leverage Measurement-Based Care for Better Patient Care, Clinician Supports, and Organizational Goals

Authors:

Elizabeth H. Connors, Susan Douglas, Amanda Jensen-Doss, Sara J. Landes, Cara C. Lewis, Bryce D. McLeod, Cameo Stanick, & Aaron R. Lyon

University of Washington affiliated authors are displayed in bold.

Subscription Required for Full Text Access

Published: July 2020

Read the full text in the subscription access journal Administration and Policy in Mental Health and Mental Health Services Research

Integration of mental health screening and treatment into cystic fibrosis clinics: Evaluation of initial implementation in 84 programs across the United States

Authors:

Alexandra L. Quittner, Janice Abbott, Saida Hussain, Thida Ong, Ahmet Uluer, Sarah Hempstead, Paula Lomas, & Beth Smith

University of Washington affiliated authors are displayed in bold.

Subscription Required for Full Text Access

Published: July 2020

Read the full text in the subscription access journal Pediatric Pulmonology

✪ Implementation Determinants and Outcomes of a Technology-Enabled Service Targeting Suicide Risk in High Schools: Mixed Methods Study

Authors:

Molly Adrian, Jessica Coifman, Michael D. Pullmann, Jennifer B. Blossom, Casey Chandler, Glen Coppersmith, Paul Thompson, & Aaron R. Lyon

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: July 2020

Read the full text in the open access journal JMIR Mental Health

Abstract:

Background

Technology-enabled services (TESs), which integrate human service and digital components, are popular strategies to increase the reach and impact of mental health interventions, but large-scale implementation of TESs has lagged behind their potential.

Objective

This study applied a mixed qualitative and quantitative approach to gather input from multiple key user groups (students and educators) and to understand the factors that support successful implementation (implementation determinants) and implementation outcomes of a TES for universal screening, ongoing monitoring, and support for suicide risk management in the school setting.

Methods

A total of 111 students in the 9th to 12th grade completed measures regarding implementation outcomes (acceptability, feasibility, and appropriateness) via an open-ended survey. A total of 9 school personnel (school-based mental health clinicians, nurses, and administrators) completed laboratory-based usability testing of a dashboard tracking the suicide risk of students, quantitative measures, and qualitative interviews to understand key implementation outcomes and determinants. School personnel were presented with a series of scenarios and common tasks focused on the basic features and functions of the dashboard. Directed content analysis based on the Consolidated Framework for Implementation Research was used to extract multilevel determinants (ie, the barriers or facilitators at the levels of the outer setting, inner setting, individuals, intervention, and implementation process) related to positive implementation outcomes of the TES.

Results

Overarching themes related to implementation determinants and outcomes suggest that both student and school personnel users view TESs for suicide prevention as moderately feasible and acceptable based on the Acceptability of Intervention Measure and Feasibility of Intervention Measure and as needing improvements in usability based on the System Usability Scale. Qualitative results suggest that students and school personnel view passive data collection based on social media data as a relative advantage to the current system; however, the findings indicate that the TES and the school setting need to address issues of privacy, integration into existing workflows and communication patterns, and options for individualization for student-centered care.

Conclusions

Innovative suicide prevention strategies that rely on passive data collection in the school context are a promising and appealing idea. Usability testing identified key issues for revision to facilitate widespread implementation.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ An implementation study of oral and blood‐based HIV self‐testing and linkage to care among men in rural and peri‐urban KwaZulu‐Natal, South Africa

Authors:

Adrienne E. Shapiro, Alastair van Heerden, Meighan Krows, Kombi Sausi, Nsika Sithole, Torin T. Schaafsma, Olivier Koole, Heidi van Rooyen, Connie L. Celum, & Ruanne V. Barnabas

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: June 2020

Read the full text in the open access Journal of the International AIDS Society

Abstract:

Introduction

In South Africa, HIV‐infected men are less likely than women to test and know their status (the first UNAIDS “90‐90‐90” target), and men have worse outcomes across the HIV care cascade. HIV self‐testing (HIVST) may address this testing disparity but questions remain over the most effective distribution strategy and linkage following a positive test result. We implemented a men‐focused HIVST distribution programme to evaluate components contributing to participation and retention.

Methods

We conducted an implementation study of multi‐venue HIVST kit distribution in rural and peri‐urban KwaZulu‐Natal (KZN), South Africa. We distributed HIVST kits at community points, workplaces and social venues for on site or take‐home use. Clients could choose blood‐based or oral‐fluid‐based HIVST kits and elect to watch an in‐person or video demonstration. We provided a USD2 incentive to facilitate reporting test results by phone or SMS. Persons with reactive HIVST results were provided immediate confirmatory tests (if used HIVST on site) or were referred for confirmatory testing (if took HIVST off site) and linkage to care for ART initiation. We describe the testing and linkage cascade in this sample and describe predictors of reactive HIVST results and linkage.

Results

Between July and November 2018, we distributed 4496 HIVST kits in two regions of KZN (96% to men, median age 28 (IQR 23 to 35). Most participants (58%) chose blood‐based HIVST and 42% chose oral‐swab kits. 11% of men were testing for the first time. A total of 3902 (83%) of testers reported their test result to the study team, with 314 (8%) screening positive for HIV. Among 274 men with reactive HIVST results, 68% linked to ART; no significant predictors of linkage were identified. 10% of kit users reported they would prefer a different type (oral vs. blood) of kit for repeat testing than the type they used.

Conclusions

HIVST is acceptable to men and rapid distribution with optional testing support is feasible in rural and peri‐urban settings. HIVST kits successfully reached younger men and identified undetected infections. Both oral and blood‐based HIVST were selected. Scaling up HIVST distribution and guidance may increase the number of first‐time testers among men and help achieve the first UNAIDS “90” for men in South Africa.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Community-based postpartum contraceptive counseling in rural Nepal: a mixed-methods evaluation

Authors:

Wan-Ju Wu, Aparna Tiwari, Nandini Choudhury, Indira Basnett, Rita Bhatt, David Citrin, Scott Halliday, Lal Kunwar, Duncan Maru, Isha Nirola, Sachit Pandey, Hari Jung Rayamazi, Sabitri Sapkota, Sita Saud, Aradhana Thapa, Alisa Goldberg, & Sheela Maru

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: June 2020

Read the full text in the open access journal Sexual and Reproductive Health Matters

Abstract:

Unmet need for postpartum contraception in rural Nepal remains high and expanding access to sexual and reproductive healthcare is essential to achieving universal healthcare. We evaluated the impact of an integrated intervention that employed community health workers aided by mobile technology to deliver patient-centred, home-based antenatal and postnatal counseling on postpartum modern contraceptive use. This was a pre–post-intervention study in seven village wards in a single municipality in rural Nepal.

The primary outcome was modern contraceptive use among recently postpartum women. We performed a multivariable logistic regression to examine contraceptive use among postpartum women pre- and one-year post-intervention. We conducted qualitative interviews to explore the implementation process. There were 445 postpartum women in the pre-intervention group and 508 in the post-intervention group.

Modern contraceptive use increased from 29% pre-intervention to 46% post-intervention (p < 0.0001). Adjusting for age, caste, and household expenditure, time since delivery and sex of child in the index pregnancy, postpartum women one-year post-intervention had twice the odds (OR 2.3; CI 1.7, 3.1; p < 0.0001) of using a modern contraceptive method as compared to pre-intervention. Factors at the individual, family, and systems level influenced women’s contraceptive decisions. The intervention contributed to increasing contraceptive use through knowledge transfer, demand generation, referrals to healthcare facilities, and follow-up.

A community-based, patient-centered contraceptive counseling intervention supported by mobile technology and integrated into longitudinal care delivered by community health workers appears to be an effective strategy for improving uptake of modern contraception among postpartum women in rural Nepal.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**