Audit and Feedback Archives - Implementation Science at UW

✪ Cost of an Audit and Feedback intervention to increase uptake of maternal and child health services in Mozambique’s Primary health care

Abstract:

Introduction

Despite high coverage of facility-based maternal and child health (MCH) services in Mozambique, preventable maternal and neonatal deaths remain among the highest globally. To address this, the Integrated District Evidence-to-Action (IDEAs) program was implemented from 2016 to 2020 in central Mozambique as a facility-level Audit and Feedback (A&F) strategy. IDEAs adapted traditional A&F into three components: 1) biannual facility readiness assessments, 2) A&F meetings to develop action plans, and 3) supportive supervision with funding for implementation. This study estimates total and component-specific implementation costs of IDEAs and compares them to average district health budgets.

Methods

Costs were estimated from a payer perspective following CHEERS guidelines, using both Gross and Microcosting methods. Costs were annualized over five years (2016–2020) with a 3% discount rate and are reported in 2020 USD. Staff time estimates were based on program staff consultations and national salary data. Outcomes include average cost per district and total annual cost by intervention component. We conducted a deterministic sensitivity analysis to identify cost drivers with the most significant impact on the overall cost.

Results

The IDEAs programs total (2016–2020) incurred total costs were $2,224,341 (2020 USD), with $1,702,648 (76.5%) for recurrent expenses and $495,323 (22.3%) for Capital expenses. Average annual district-level costs were $41,967, including A&F meetings ($10,893; 26.0%), Capital expenditures ($8255; 19.7%), targeted support ($9323; 22.2%) which includes $5700 (13.6%) for district-level cash transfers and $3623 (8.6%) for facility supervision, and readiness assessments ($6351; 15.1%).Per diems accounted for 52.0% of A&F meeting costs. IDEAs represented approximately 6.6% of the average district health budget. Implementation involved 33 participants with 2560 h yearly per district. Capital expenses were highly sensitive to when applying a standard 25% increase or decrease compared to other cost categories.

Conclusion

Our findings offer practical guidance for district-level planners to adapt and scale IDEAs. Policymakers and donors should integrate recurring costs into budgets and explore cost-saving strategies like virtual tools or streamlined meetings to improve sustainability. Future research must assess long-term integration into government programs and test alternative approaches across diverse low-resource settings to guide scaling.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Exploring fidelity and its influence on effectiveness in an audit and feedback strategy implemented in a low-resource setting: an application of regression modeling

Abstract:

Background

Assessing implementation fidelity—the degree to which a program is implemented as intended—is essential to understand whether poor outcomes are due to implementation problems or the design of an intervention. Few studies in health research have documented the association between implementation fidelity and effectiveness. The Integrated District Evidence-to-Action (IDEAs) is a multicomponent audit and feedback strategy designed to improve the implementation of maternal and child clinical guidelines in Mozambique. In a previous study, we found mixed results of IDEAs effectiveness. The objective of the present study is to understand how implementation fidelity may have influenced the effectiveness of the strategy.

Methods

IDEAs was implemented in 154 health facilities across 12 districts in Manica and Sofala provinces in Mozambique between 2016 and 2020. We used the conceptual framework for implementation fidelity to guide descriptive analysis of IDEAs adherence. Regression modeling was used to study patterns of the direction of association between measures of fidelity and effectiveness for ten service delivery outcomes and five service readiness outcomes.

Results

We describe adherence on 15 measures of fidelity, of which 12 had high fidelity. Poor fidelity was found in conducting facility service readiness assessments and completing micro-interventions from action plans. Service delivery measures tended to be positively associated with participation and degree of micro-intervention completion and negatively associated with a higher number of action plans elaborated by participating teams. For the service readiness outcomes, delivery of essential care was positively associated with participation and micro-intervention completion, and staff availability was negatively associated with supervision.

Conclusion

Participation in audit and feedback meetings, the number of action plans elaborated, and the degree of completion of micro-interventions seem to be related to the effectiveness results. IDEAs should be adapted to reduce the number of action plans elaborated and promote better micro-intervention completion. Additionally, combining audit and feedback strategies with other strategies might enhance effectiveness in service outcomes. This study examines how to analyze the link between fidelity and effectiveness of a strategy to inform better design and recommend context-specific improvements.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ A pragmatic approach to estimating the cost to deliver and participate in implementation strategies

Abstract:

The Healing Experiences of Adversity Among Latinos (HEALthy4You; H4Y) study was a multi-sector partnership between an academic research institution, a Federally Qualified Health Center (FQHC), and a multi-sector collective impact coalition focused on childhood obesity prevention. The goal of HEALthy4You was to develop community-centered and culturally appropriate precision interventions within FQHCs for Latino families to address predictors of adverse child experiences and treat childhood obesity. A multidisciplinary and multi-sector research, clinical, and community team (N = 29) was formed in September 2020 to co-design the study, which launched in June 2022. The team utilized a co-creation approach combined with the Exploration, Preparation, Implementation, and Sustainment framework to facilitate a collaborative design process. We conducted an internal and retrospective process evaluation in March 2023 to identify antecedents and situational factors associated with project formation, with a focus on understanding tensions and challenges with a broad partnership structure. We outline the team's co-creation process and describe internal challenges and pitfalls that emerged when developing the project. We sought to better understand the impact of differing perspectives, priorities, and goals between disciplines, sectors, and roles; differing approaches to evidence and evidence production; and team strategies to mitigate and manage competing pressures and priorities. This case report describes lessons learned, intending to share insights to support future development of best practices in project, partner, and team formation between researchers, clinicians, and community members. More specifically, these lessons could help inform community-led research endeavors between academic institutions, FQHCs, and community-based organizations (CBOs).

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Can audit and feedback improve health service readiness and delivery outcomes in a low-resource setting? Effectiveness results of the IDEAs strategy from central Mozambique

Abstract:

Limited evidence exists on audit and feedback (A&F) in low-resource contexts. We tested the Integrated District Evidence-to-Action (IDEAs), a multicomponent A&F strategy in Mozambique’s maternal and child health (MCH) services. IDEAs include A&F meetings, readiness assessments, and facility support. We report the effectiveness results. IDEAs were implemented in 2016–2020 across two provinces, 12 districts, and 154 primary health facilities in Mozambique. We assessed 1) ten service delivery outcomes across antenatal, maternity, postpartum, childcare, and reproductive health services and 2) five service readiness outcomes (medicines, infrastructure, equipment, care, and staffing availability). We used propensity score matching to minimize bias and a controlled interrupted time series with a negative binomial mixed effects model for service delivery analysis, presenting incidence rate ratios (IRR) with 95% confidence intervals (95% CI). For service readiness outcomes, we created composite scores for each domain and a difference-in-difference analysis using an ordinal mixed effects model, reporting odds ratios (OR) and 95% CI. Significant associations were found with first at-risk child appointments (IRR = 1.06 [1.04, 1.07]), first polymerase chain reaction tests for HIV-exposed children (IRR = 1.02 [1.01, 1.03]), new contraceptive users (IRR = 0.95 [0.94, 0.96]), women starting long-lasting contraceptives (IRR = 0.94 [0.93, 0.95]), availability of infrastructure (OR = 5.84 [1.32, 25.88]) and essential care (OR = 0.13 [0.03, 0.54]). No significant associations were found between IDEAs and six of 10 service delivery outcomes (women with a fourth dose of preventive malaria treatment; women protected with tetanus vaccine; women with four+ antenatal visits; deliveries with active management of the third stage of labor; first postpartum consultations and fully vaccinated children) and on medicine, equipment, and staffing availability. We observed mixed effectiveness in implementing IDEAs, with null and sub-optimal effects suggesting the need to refine and adapt strategy components to more effectively address clinical and readiness outcomes.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Implementation outcomes of the integrated district evidence to action (IDEAs) program to reduce neonatal mortality in central Mozambique: an application of the RE-AIM evaluation framework

Authors:

Aneth Dinis, Quinhas Fernandes, Bradley H Wagenaar, Sarah Gimbel, Bryan J Weiner, Grace John-Stewart, Ermyas Birru, Stephen Gloyd, Ruth Etzioni, Dorlim Uetela, Isaías Ramiro, Artur Gremu, Orvalho Augusto, Stélio Tembe, Jaime L Mário, Jalilo E Chinai, Alfredo F Covele, Cassimo M Sáide, Nélia Manaca & Kenneth Sherr

^{University of Washington affiliated authors are displayed in bold.}

✪ Open Access

Published: February 2024

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

Scarce evidence exists on audit and feedback implementation processes in low-resource health systems. The Integrated District Evidence to Action (IDEAs) is a multi-component audit and feedback strategy designed to improve the implementation of maternal and child guidelines in Mozambique. We report IDEAs implementation outcomes.

Methods

IDEAs was implemented in 154 health facilities across 12 districts in Manica and Sofala provinces between 2016 and 2020 and evaluated using a quasi-experimental design guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. Reach is the proportion of pregnant women attending IDEAs facilities. Adoption is the proportion of facilities initiating audit and feedback meetings. Implementation is the fidelity to the strategy components, including readiness assessments, meetings (frequency, participation, action plan development), and targeted financial support and supervision. Maintenance is the sustainment at 12, 24, and 54 months.

Results

Across both provinces, 56% of facilities were exposed to IDEAs (target 57%). Sixty-nine and 73% of pregnant women attended those facilities’ first and fourth antenatal consultations (target 70%). All facilities adopted the intervention. 99% of the expected meetings occurred with an average interval of 5.9 out of 6 months. Participation of maternal and child managers was high, with 3076 attending meetings, of which 64% were from the facility, 29% from the district, and 7% from the province level. 97% of expected action plans were created, and 41 specific problems were identified. “Weak diagnosis or management of obstetric complications” was identified as the main problem, and “actions to reinforce norms and protocols” was the dominant subcategory of micro-interventions selected. Fidelity to semiannual readiness assessments was low (52% of expected facilities), and in completing micro-interventions (17% were completed). Ninety-six and 95% of facilities sustained the intervention at 12 and 24 months, respectively, and 71% had completed nine cycles at 54 months.

Conclusion

Maternal and child managers can lead audit and feedback processes in primary health care in Mozambique with high reach, adoption, and maintenance. The IDEAs strategy should be adapted to promote higher fidelity around implementing action plans and conducting readiness assessments. Adding effectiveness to these findings will help to inform strategy scale-up.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Strengthening integrated depression services within routine primary health care using the RE-AIM framework in South Africa

Authors:

Inge Petersen, Christopher G. Kemp, Deepa Rao, Bradley H. Wagenaar, Max Bachmann, Kenneth Sherr, Tasneem Kathree, Zamasomi Luvuno, André Van Rensburg, Sithabisile Gugulethu Gigaba, Londiwe Mthethwa, Merridy Grant, One Selohilwe, Nikiwe Hongo, Gillian Faris, Christy-Joy Ras, Lara Fairall, Sanah Bucibo, & Arvin Bhana

^{University of Washington affiliated authors are displayed in bold.}

✪ Open Access

Published: November 2023

Read the full text in the open access journal PLOS Global Public Health

Abstract:

Integration of mental health into routine primary health care (PHC) services in low-and middle-income countries is globally accepted to improve health outcomes of other conditions and narrow the mental health treatment gap. Yet implementation remains a challenge. The aim of this study was to identify implementation strategies that improve implementation outcomes of an evidence-based depression care collaborative implementation model integrated with routine PHC clinic services in South Africa. An iterative, quasi-experimental, observational implementation research design, incorporating the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework, was applied to evaluate implementation outcomes of a strengthened package of implementation strategies (stage two) compared with an initial evaluation of the model (stage one). The first stage package was implemented and evaluated in 10 PHC clinics and the second stage strengthened package in 19 PHC clinics (inclusive of the initial 10 clinics) in one resource-scarce district in the province of KwaZulu-Natal, South Africa. Diagnosed service users were more likely to be referred for counselling treatment in the second stage compared with stage one (OR 23.15, SE = 18.03, z = 4.04, 95%CI [5.03–106.49], p < .001). Training in and use of a validated, mandated mental health screening tool, including on-site educational outreach and technical support visits, was an important promoter of nurse-level diagnosis rates (OR 3.75, 95% CI [1.19, 11.80], p = 0.02). Nurses who perceived the integrated care model as acceptable were also more likely to successfully diagnose patients (OR 2.57, 95% CI [1.03–6.40], p = 0.043). Consistent availability of a clinic counsellor was associated with a greater probability of referral (OR 5.9, 95%CI [1.29–27.75], p = 0.022). Treatment uptake among referred service users remained a concern across both stages, with inconsistent co-located counselling services associated with poor uptake. The importance of implementation research for strengthening implementation strategies along the cascade of care for integrating depression care within routine PHC services is highlighted.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Applying the Consolidated Framework for Implementation Research to Identify Implementation Determinants for the Integrated District Evidence-to-Action Program, Mozambique

Abstract

Introduction

The Integrated District Evidence-to-Action program is an audit and feedback intervention introduced in 2017 in Manica and Sofala provinces, Mozambique, to reduce mortality in children younger than 5 years. We describe barriers and facilitators to early-stage effectiveness of that intervention.

Method

We embedded the Consolidated Framework for Implementation Research (CFIR) into an extended case study design to inform sampling, data collection, analysis, and interpretation. We collected data in 4 districts in Manica and Sofala Provinces in November 2018. Data collection included document review, 22 in-depth individual interviews, and 2 focus group discussions (FGDs) with 19 provincial, district, and facility managers and nurses. Most participants (70.2%) were nurses and facility managers and the majority were women (87.8%). We audio-recorded all but 2 interviews and FGDs and conducted a consensus-based iterative analysis.

Results

Facilitators of effective intervention implementation included: implementation of the core intervention components of audit and feedback meetings, supportive supervision and mentorship, and small grants as originally planned; positive pressure from district managers and study nurses on health facility staff to strive for excellence; and easy access to knowledge and information about the intervention. Implementation barriers were the intervention’s lack of compatibility in not addressing the scarcity of human and financial resources and inadequate infrastructures for maternal and child health services at district and facility levels and; the intervention’s lack of adaptability in having little flexibility in the design and decision making about the use of intervention funds and data collection tools.

Discussion

Our comprehensive and systematic use of the CFIR within an extended case study design generated granular evidence on CFIR’s contribution to implementation science efforts to describe determinants of early-stage intervention implementation. It also provided baseline findings to assess subsequent implementation phases, considering similarities and differences in barriers and facilitators across study districts and facilities. Sharing preliminary findings with stakeholders promoted timely decision making about intervention implementation.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors’ Response to: Novel Data Linkage for Quality Improvement in Palliative and End-of-Life Care

✪ Implementation Strategies to Enhance the Implementation of eHealth Programs for Patients With Chronic Illnesses: Realist Systematic Review

Abstract:

Background

There is growing evidence of the positive effects of electronic health (eHealth) interventions for patients with chronic illness, but implementation of such interventions into practice is challenging. Implementation strategies that potentially impact implementation outcomes and implementation success have been identified. Which strategies are actually used in the implementation of eHealth interventions for patients with chronic illness and which ones are the most effective is unclear.

Objective

This systematic realist review aimed to summarize evidence from empirical studies regarding (1) which implementation strategies are used when implementing eHealth interventions for patients with chronic illnesses living at home, (2) implementation outcomes, and (3) the relationship between implementation strategies, implementation outcomes, and degree of implementation success.

Methods

A systematic literature search was performed in the electronic databases MEDLINE, Embase, PsycINFO, Scopus, Allied and Complementary Medicine Database, Cumulative Index to Nursing and Allied Health Literature, and Cochrane Library. Studies were included if they described implementation strategies used to support the integration of eHealth interventions into practice. Implementation strategies were categorized according to 9 categories defined by the Expert Recommendations for Implementing Change project: (1) engage consumers, (2) use evaluative and iterative strategies, (3) change infrastructure, (4) adapt and tailor to the context, (5) develop stakeholder interrelationships, (6) use financial strategies, (7) support clinicians, (8) provide interactive assistance, and (9) train and educate stakeholders. Implementation outcomes were extracted according to the implementation outcome framework by Proctor and colleagues: (1) acceptability, (2) adoption, (3) appropriateness, (4) cost, (5) feasibility, (6) fidelity, (7) penetration, and (8) sustainability. Implementation success was extracted according to the study authors’ own evaluation of implementation success in relation to the used implementation strategies.

Results

The implementation strategies management support and engagement, internal and external facilitation, training, and audit and feedback were directly related to implementation success in several studies. No clear relationship was found between the number of implementation strategies used and implementation success.

Conclusions

This is the first review examining implementation strategies, implementation outcomes, and implementation success of studies reporting the implementation of eHealth programs for patients with chronic illnesses living at home. The review indicates that internal and external facilitation, audit and feedback, management support, and training of clinicians are of importance for eHealth implementation. The review also points to the lack of eHealth studies that report implementation strategies in a comprehensive way and highlights the need to design robust studies focusing on implementation strategies in the future.

Trial Registration

PROSPERO CRD42018085539

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Assessing Levels and Correlates of Implementation of Evidence-Based Approaches for Colorectal Cancer Screening: A Cross-Sectional Study With Federally Qualified Health Centers

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Introduction

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal PLOS Global Public Health

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal PLOS Global Public Health

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal Global Health: Science and Practice

Abstract

Introduction

Method

Results

Discussion

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access Journal of Pain and Symptom Management

Authors:

Read the full text in the open access Journal of Medical Internet Research

Abstract:

Background

Objective

Methods

Results

Conclusions

Trial Registration

Authors: