Evaluation Archives - Implementation Science at UW

✪ Formative Evaluation of a Comprehensive Self-Management Intervention for Irritable Bowel Syndrome, Comorbid Anxiety, and Depression: Mixed Methods Study

Abstract:

Background

Irritable bowel syndrome (IBS) is a disorder of the gut-brain interaction that is associated with abdominal pain, altered bowel patterns, and reduced quality of life. Up to 50% of patients with IBS also report anxiety or depressive symptoms. Although effective self-management interventions exist for individuals with IBS, few have been effectively implemented, and most do not consider the unique needs of patients with comorbid IBS and anxiety or depression.

Objective

This study aimed to determine the anticipated acceptability, appropriateness, feasibility, and usability of a comprehensive self-management intervention using an implementation science and human-centered design approach among individuals with comorbid IBS and anxiety or depression and health care providers.

Methods

A convergent mixed methods design was used to elicit feedback on the comprehensive self-management intervention outline and content to identify refinement needs before testing. Patients with IBS and moderate to severe anxiety or depression and health care providers were purposefully sampled from primary care and gastroenterology settings. Participants completed semistructured interviews and surveys on anticipated acceptability, appropriateness, feasibility, and usability.

Results

Patient participants (n=12) were on average 36.8 (SD 12.2) years of age, and 42% (5/12) were currently receiving psychological therapy. Health care providers (n=14) were from primary care (n=7) and gastroenterology (n=7) settings. The mean usability scores (out of 100) were 52.5 (SD 14.5) for patients and 45.6 (SD 11.6) for providers. For patients and providers, qualitative data expanded the quantitative findings for acceptability and appropriateness. Acceptability findings were the comprehensive nature of the intervention and discussion of the gut-brain interaction. For appropriateness, participants reported that the intervention provided structure, accountability, and support. Feasibility was confirmed for patients, but there was a divergence of findings between quantitative and qualitative measures for providers. Patients focused on intervention feasibility, while providers focused on implementation feasibility in the clinic. Identified usability issues to address before implementation included the intervention delivery format, length, and lack of integration into health care settings that, if not addressed, may limit the reach of the intervention

Conclusions

Patients and health care providers found the intervention acceptable and appropriate. Several feasibility and usability issues were identified, including intervention delivery methods, length of intervention, and the best methods to implement in the clinic setting. The next steps are to refine the intervention to address the identified issues and test in a pilot study whether addressing usability issues leads to the anticipated improvements in implementation and uptake.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Realist evaluation in times of decolonising global health

✪ Process evaluation of PrEP implementation in Kenya: adaptation of practices and contextual modifications in public HIV care clinics

Abstract:

Introduction

In Africa, oral pre-exposure prophylaxis (PrEP) is largely provided via over-burdened public HIV care clinics. Successfully incorporating PrEP services into these clinics may require adaptations to practices outlined in national implementation guidelines and modifications to routine existing service delivery. We aimed to describe adaptations made by public HIV clinics in Kenya to integrate PrEP delivery into existing services.

Methods

The Partners Scale-Up Project aimed to catalyze integration of PrEP in 25 public HIV care clinics. Between May and December 2018, we conducted qualitative interviews with health providers and documented clinic observations in technical assistance (TA) reports to understand the process of PrEP service integration. We analyzed 36 health provider interview transcripts and 25 TA reports to identify clinic-level adaptations to activities outlined in Kenyan Ministry of Health PrEP guidelines and modifications made to existing service delivery practices to successfully incorporate PrEP services. Identified adaptations were reported using the expanded framework for reporting adaptations and modifications (FRAME).

Results

All clinics (n = 25) performed HIV testing, HIV risk assessment, PrEP education and adherence counseling as stipulated in the guidelines. Most clinics initiated clients on PrEP without creatinine testing if otherwise healthy. While monthly refill appointments are recommended, a majority of clinics issued PrEP users two to three months of pills at a time. Clinics also implemented practices that had not been specified in the guidelines including incorporating PrEP-related topics into routine health talks, calling clients with missed PrEP appointments, discussing PrEP service delivery in regular staff meetings, ‘fast-tracking’ PrEP clients and dispensing PrEP in clinic rooms rather than at clinic-based pharmacies. PrEP initiation numbers were highest among clinics that did not require creatinine testing, conducted peer on-the-job PrEP training and those that discussed PrEP delivery in their routine meetings. Above-average continuation was observed among clinics that discussed PrEP in their routine meetings, dispensed PrEP in clinic rooms and offered PrEP at non-regular hours.

Conclusions

Health providers in public HIV care clinics instituted practices and made innovative adaptations to PrEP delivery to reduce barriers for clients and staff. Encouraging clinic level adaptations to national implementation guidelines will facilitate scale-up of PrEP delivery.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide

Abstract:

Background

Suicide rates continue to rise across the United States, galvanizing the need for increased suicide prevention and intervention efforts. The Zero Suicide (ZS) model was developed in response to this need and highlights four key clinical functions of high-quality health care for patients at risk of suicide. The goal of this quality improvement study was to understand how six large health care systems operationalized practices to support these functions—identification, engagement, treatment and care transitions.

Methods

Using a key informant interview guide and data collection template, researchers who were embedded in each health care system cataloged and summarized current and future practices supporting ZS, including, (1) the function addressed; (2) a description of practice intent and mechanism of intervention; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented and/or measured. Normalization process theory (NPT), an implementation evaluation framework, was applied to help understand how ZS had been operationalized in routine clinical practices and, specifically, what ZS practices were described by key informants (coherence), the current state of norms/conventions supporting these practices (cognitive participation), how health care teams performed these practices (collective action), and whether/how practices were measured when they occurred (reflexive monitoring).

Results

The most well-defined and consistently measured ZS practices (current and future) focused on the identification of patients at high risk of suicide. Stakeholders also described numerous engagement and treatment practices, and some practices intended to support care transitions. However, few engagement and transition practices were systematically measured, and few treatment practices were designed specifically for patients at risk of suicide.

Conclusions

The findings from this study will support large-scale evaluation of the effectiveness of ZS implementation and inform recommendations for implementation of high-quality suicide-related care in health care systems nationwide.

Plain Language Summary

Many individuals see a health care provider prior to death by suicide, therefore health care organizations have an important role to play in suicide prevention. The Zero Suicide model is designed to address four key functions of high-quality care for patients at risk of suicide: (1) identification of suicide risk via routine screening/assessment practices, (2) engagement of patients at risk in care, (3) effective treatment, and (4) care transition support, particularly after hospitalizations for suicide attempts. Researchers embedded in six large health care systems, together caring for nearly 11.5 million patients, are evaluating the effectiveness of the Zero Suicide model for suicide prevention. This evaluation focused on understanding how these systems had implemented clinical practices supporting Zero Suicide. Researchers collected qualitative data from providers, administrators, and support staff in each system who were responsible for implementation of practices supporting Zero Suicide. Normalization process theory, an implementation evaluation framework, was applied following data collection to: (A) help researchers catalog all Zero Suicide practices described, (B) describe the norms/conventions supporting these practices, (C) describe how health care teams were performing these practices, and (D) describe how practices were being measured. The findings from this evaluation will be vital for measuring the effectiveness of different Zero Suicide practices. This work will also provide a blueprint to help health care leaders, providers, and other stakeholders “normalize” new and existing suicide prevention practices in their own organizations.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ How to evaluate the implementation of complex health programmes in low-income settings: the approach of the Gavi Full Country Evaluations

Abstract:

Vaccination, like most other public health services, relies on a complex package of intervention components, functioning systems and committed actors to achieve universal coverage. Despite significant investment in immunization programmes, national coverage trends have slowed and equity gaps have grown. This paper describes the design and implementation of the Gavi Full Country Evaluations, a multi-country, prospective, mixed-methods approach whose goal was to monitor and evaluate processes, inputs, outputs and outcomes of immunization programmes in Bangladesh, Mozambique, Uganda and Zambia.

We implemented the Full Country Evaluations from 2013 to 2018 with the goal of identifying the drivers of immunization programme improvement to support programme implementation and increase equitable immunization coverage. The framework supported methodological and paradigmatic flexibility to respond to a broad range of evaluation and implementation research questions at global, national and cross-country levels, but was primarily underpinned by a focus on evaluating processes and identifying the root causes of implementation breakdowns. Process evaluation was driven by theories of change for each Gavi funding stream (e.g. Health Systems Strengthening) or activity, ranging from global policy development to district-level programme implementation.

Mixing of methods increased in relevance and rigour over time as we learned to build multiple methods into increasingly tailored evaluation questions. Evaluation teams in country-based research institutes increasingly strengthened their level of embeddedness with immunization programmes as the emphasis shifted over time to focus more heavily on the use of findings for programme learning and adaptation. Based on our experiences implementing this approach, we recommend it for the evaluation of other complex interventions, health programmes or development assistance.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Mentorship and coaching to support strengthening healthcare systems: lessons learned across the five Population Health Implementation and Training partnership projects in sub-Saharan Africa

Abstract:

Background

Despite global efforts to increase health workforce capacity through training and guidelines, challenges remain in bridging the gap between knowledge and quality clinical practice and addressing health system deficiencies preventing health workers from providing high quality care. In many developing countries, supervision activities focus on data collection, auditing and report completion rather than catalyzing learning and supporting system quality improvement. To address this gap, mentorship and coaching interventions were implemented in projects in five African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) as components of health systems strengthening (HSS) strategies funded through the Doris Duke Charitable Foundation’s African Health Initiative. We report on lessons learned from a cross-country evaluation.

Methods

The evaluation was designed based on a conceptual model derived from the project-specific interventions. Semi-structured interviews were administered to key informants to capture data in six categories: 1) mentorship and coaching goals, 2) selection and training of mentors and coaches, 3) integration with the existing systems, 4) monitoring and evaluation, 5) reported outcomes, and 6) challenges and successes. A review of project-published articles and technical reports from the individual projects supplemented interview information.

Results

Although there was heterogeneity in the approaches to mentorship and coaching and targeted areas of the country projects, all led to improvements in core health system areas, including quality of clinical care, data-driven decision making, leadership and accountability, and staff satisfaction. Adaptation of approaches to reflect local context encouraged their adoption and improved their effectiveness and sustainability.

Conclusion

We found that incorporating mentorship and coaching activities into HSS strategies was associated with improvements in quality of care and health systems, and mentorship and coaching represents an important component of HSS activities designed to improve not just coverage, but even further effective coverage, in achieving Universal Health Care.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Formative Research

Abstract:

Background

Objective

Methods

Results

Conclusions

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal, The International Journal of Health Planning and Management

Authors:

✪ Open Access

Read the full text in the open access Journal of the International AIDS Society

Abstract:

Introduction

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Methods

Results

Conclusions

Plain Language Summary

Authors:

✪ Open Access

Read the full text in the open access journal Health Policy and Planning

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

Methods

Results

Conclusion