Skip to content
Implementation Science at UW

Archives: UW Articles

Measuring health systems strength and its impact: experiences from the African Health Initiative

Authors:

Kenneth Sherr, Quinhas Fernandes, Almamy M. Kanté, Ayaga Bawah, Jeanine Condo, Wilbroad Mutale and the AHI PHIT Partnership Collaborative

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: December 2017

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

Health systems are essential platforms for accessible, quality health services, and population health improvements. Global health initiatives have dramatically increased health resources; however, funding to strengthen health systems has not increased commensurately, partially due to concerns about health system complexity and evidence gaps demonstrating health outcome improvements. In 2009, the African Health Initiative of the Doris Duke Charitable Foundation began supporting Population Health Implementation and Training Partnership projects in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to catalyze significant advances in strengthening health systems. This manuscript reflects on the experience of establishing an evaluation framework to measure health systems strength, and associate measures with health outcomes, as part of this Initiative.

Methods

Using the World Health Organization’s health systems building block framework, the Partnerships present novel approaches to measure health systems building blocks and summarize data across and within building blocks to facilitate analytic procedures. Three Partnerships developed summary measures spanning the building blocks using principal component analysis (Ghana and Tanzania) or the balanced scorecard (Zambia). Other Partnerships developed summary measures to simplify multiple indicators within individual building blocks, including health information systems (Mozambique), and service delivery (Rwanda). At the end of the project intervention period, one to two key informants from each Partnership’s leadership team were asked to list – in rank order – the importance of the six building blocks in relation to their intervention.

Results

Though there were differences across Partnerships, service delivery and information systems were reported to be the most common focus of interventions, followed by health workforce and leadership and governance. Medical products, vaccines and technologies, and health financing, were the building blocks reported to be of lower focus.

Conclusion

The African Health Initiative experience furthers the science of evaluation for health systems strengthening, highlighting areas for further methodological development – including the development of valid, feasible measures sensitive to interventions in multiple contexts (particularly in leadership and governance) and describing interactions across building blocks; in developing summary statistics to facilitate testing intervention effects on health systems and associations with health status; and designing appropriate analytic models for complex, multi-level open health systems.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

Tackling the hard problems: Implementation experience and lessons learned in newborn health from the African Health Initiative

Authors:

Hema Magge, Roma Chilengi, Elizabeth F. Jackson, Bradley H. Wagenaar, Almamy Malick Kante and the AHI PHIT Partnership Collaborative

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: December 2017

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

The Doris Duke Charitable Foundation’s African Health Initiative supported the implementation of Population Health Implementation and Training (PHIT) Partnership health system strengthening interventions in designated areas of five countries: Ghana, Mozambique, Rwanda, Tanzania, and Zambia. All PHIT programs included health system strengthening interventions with child health outcomes from the outset, but all increasingly recognized the need to increase focus to improve health and outcomes in the first month of life. This paper uses a case study approach to describe interventions implemented in newborn health, compare approaches, and identify lessons learned across the programs’ collective implementation experience.

Methods

Case studies were built using quantitative and qualitative methods, applying the World Health Organization Health Systems Strengthening Framework, and maternal, newborn and child health continuum of care framework. We identified the following five primary themes in health systems strengthening intervention strategies used to target improvement in newborn health, which were incorporated by all PHIT projects with varying results: health service delivery at the community level (Tanzania), combining community and health facility level interventions (Zambia), participatory information feedback and clinical training (Ghana), performance review and enhancement (Mozambique), and integrated clinical and system-level improvement (Rwanda), and used individual case studies to illustrate each of these themes.

Results

Tanzania and Zambia included significant community-based components, including mobilization and sensitization for increased uptake of essential services, while Ghana, Mozambique, and Rwanda focused more efforts on improving the quality of services delivered once a patient enters a health facility. All countries included aspects that improved communication across levels of the health system, whether through district-wide data sharing and peer learning networks in Mozambique and Rwanda, or improved referral processes and systems in Tanzania, Zambia, and Ghana.

Conclusion

Key lessons learned include the importance of focusing intervention components on addressing drivers of neonatal mortality across the maternal and newborn care continuum at all levels of the health system, matching efforts to improve service utilization with provision of high quality facility-based services, and the critical role of leadership to catalyze improvements in newborn health.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research (CFIR): results from the African Health Initiative

Authors:

Sarah Gimbel, Moses Mwanza, Marie Paul Nisingizwe, Cathy Michel, Lisa Hirschhorn and the AHI PHIT Partnership Collaborative

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: December 2017

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

High-quality data are critical to inform, monitor and manage health programs. Over the seven-year African Health Initiative of the Doris Duke Charitable Foundation, three of the five Population Health Implementation and Training (PHIT) partnership projects in Mozambique, Rwanda, and Zambia introduced strategies to improve the quality and evaluation of routinely-collected data at the primary health care level, and stimulate its use in evidence-based decision-making. Using the Consolidated Framework for Implementation Research (CFIR) as a guide, this paper: 1) describes and categorizes data quality assessment and improvement activities of the projects, and 2) identifies core intervention components and implementation strategy adaptations introduced to improve data quality in each setting.

Methods

The CFIR was adapted through a qualitative theme reduction process involving discussions with key informants from each project, who identified two domains and ten constructs most relevant to the study aim of describing and comparing each country’s data quality assessment approach and implementation process. Data were collected on each project’s data quality improvement strategies, activities implemented, and results via a semi-structured questionnaire with closed and open-ended items administered to health management information systems leads in each country, with complementary data abstraction from project reports.

Results

Across the three projects, intervention components that aligned with user priorities and government systems were perceived to be relatively advantageous, and more readily adapted and adopted. Activities that both assessed and improved data quality (including data quality assessments, mentorship and supportive supervision, establishment and/or strengthening of electronic medical record systems), received higher ranking scores from respondents.

Conclusion

Our findings suggest that, at a minimum, successful data quality improvement efforts should include routine audits linked to ongoing, on-the-job mentoring at the point of service. This pairing of interventions engages health workers in data collection, cleaning, and analysis of real-world data, and thus provides important skills building with on-site mentoring. The effect of these core components is strengthened by performance review meetings that unify multiple health system levels (provincial, district, facility, and community) to assess data quality, highlight areas of weakness, and plan improvements.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Stepped-Wedge Cluster Randomized Controlled Trial to Promote Option B+ Retention in Central Mozambique

Authors:

James T. Pfeiffer; Manuel Napúa; Bradley H. Wagenaar; Falume Chale; Roxanne Hoek; Mark Micek; João Manuel; Cathy Michel; Jessica Greenberg Cowan; James F. Cowan; Sarah Gimbel; Kenneth Sherr; Stephen Gloyd; Rachel R. Chapman

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: November 2017

Read the full text in the subscription access JAIDS: Journal of Acquired Immune Deficiency Syndromes

✪ Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping

Authors:

Byron J. Powell, Cameo F. Stanick, Heather M. Halko, Caitlin N. Dorsey, Bryan J. Weiner, Melanie A. Barwick, Laura J. Damschroder, Michel Wensing, Luke Wolfenden, and Cara C. Lewis

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: October 2017

Read the full text in the open access journal Implementation Science

Abstract:

Background

Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria’s clarity and importance.

Methods

Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data.

Findings

The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented.

Conclusions

This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

Intentional research design in implementation science: Implications for the use of nomothetic and idiographic assessment

Authors:

Aaron R. Lyon, Elizabeth Connors, Amanda Jensen-Doss, Sara J. Landes, Cara C. Lewis, Bryce D. McLeod, Christopher Rutt, Cameo Stanick, Bryan J. Weiner

University of Washington affiliated authors are displayed in bold.

Subscription Required for Full Text Access

Published: September 2017

Read the full text in the subscription access journal Translational Behavioral Medicine

✪ Psychometric assessment of three newly developed implementation outcome measures

Authors:

Bryan J. Weiner, Cara C. Lewis, Cameo Stanick, Byron J. Powell, Caitlin N. Dorsey, Alecia S. Clary, Marcella H. Boynton and Heather Halko

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: August 2017

Read the full text in the open access journal Implementation Science

Abstract:

Background

Implementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).

Methods

Thirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change.

Results

All but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α’s from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions.

Conclusions

The AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Using Practice Facilitation to Increase Rates of Colorectal Cancer Screening in Community Health Centers, North Carolina, 2012-2013: Feasibility, Facilitators, and Barriers

Authors:

Bryan J. Weiner, Catherine L. Rohweder, Jennifer E. Scott; Randall Teal, Alecia Slade, Allison M. Deal, Naima Jihad, & Marti Wolf

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: August 2017

Read the full text in the open access journal Preventing Chronic Disease

Abstract:

Introduction

Practice facilitation involves trained individuals working with practice staff to conduct quality improvement activities and support delivery of evidence-based clinical services. We examined the feasibility of using practice facilitation to assist federally qualified health centers (FQHCs) to increase colorectal cancer screening rates in North Carolina.

Methods

The intervention consisted of 12 months of facilitation in 3 FQHCs. We conducted chart audits to obtain data on changes in documented recommendation for colorectal cancer screening and completed screening. Key informant interviews provided qualitative data on barriers to and facilitators of implementing office systems.

Results

Overall, the percentage of eligible patients with a documented colorectal cancer screening recommendation increased from 15% to 29% (P < .001). The percentage of patients up to date with colorectal cancer screening rose from 23% to 34% (P = .03). Key informants in all 3 clinics said the implementation support from the practice facilitator was critical for initiating or improving office systems and that modifying the electronic medical record was the biggest challenge and most time-consuming aspect of implementing office systems changes. Other barriers were staff turnover and reluctance on the part of local gastroenterology practices to perform free or low-cost diagnostic colonoscopies for uninsured or underinsured patients.

Conclusion

Practice facilitation is a feasible, acceptable, and promising approach for supporting universal colorectal cancer screening in FQHCs. A larger-scale study is warranted.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Pre/post evaluation of a pilot prevention with positives training program for healthcare providers in North West Province, Republic of South Africa

Authors:

Christopher G. Kemp, Julia de Kadt, Erushka Pillay, Jennifer M. Gilvydis, Evasen Naidoo, Jessica Grignon and Marcia R. Weaver

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: May 2017

Read the full text in the open access journal BMC Health Services Research

Abstract:

Background

Prevention interventions for people living with HIV/AIDS are an important component of HIV programs. We report the results of a pilot evaluation of a four-hour, clinic-based training for healthcare providers in South Africa on HIV prevention assessments and messages. This pre/post pilot evaluation examined whether the training was associated with providers delivering more prevention messages.

Methods

Seventy providers were trained at four public primary care clinics with a high volume of HIV patients. Pre- and post-training patient exit surveys were conducted using Audio-Computer Assisted Structured Interviews. Seven provider appropriate messaging outcomes and one summary provider outcome were compared pre- and post-training using Poisson regression.

Results

Four hundred fifty-nine patients pre-training and 405 post-training with known HIV status were interviewed, including 175 and 176 HIV positive patients respectively. Among HIV positive patients, delivery of all appropriate messages by providers declined post-training. The summary outcome decreased from 56 to 50%; adjusted rate ratio 0.92 (95% CI = 0.87–0.97). Sensitivity analyses adjusting for training coverage and time since training detected fewer declines. Among HIV negative patients the summary score was stable at 32% pre- and post-training; adjusted rate ratio 1.05 (95% CI = 0.98–1.12).

Conclusions

Surprisingly, this training was associated with a decrease in prevention messages delivered to HIV positive patients by providers. Limited training coverage and delays between training and post-training survey may partially account for this apparent decrease. A more targeted approach to prevention messages may be more effective.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**