Skip to content
Implementation Science at UW

Keywords: Cara C. Lewis

✪ From Classification to Causality: Advancing Understanding of Mechanisms of Change in Implementation Science

Authors:

Cara C. Lewis, Predrag Klasnja, Byron J. Powell, Aaron R. Lyon, Leah Tuzzio, Salene Jones, Callie Walsh-Bailey and Bryan J. Weiner

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: May 2018

Read the full text in the open access journal Frontiers in Public Health

Abstract:

Background

The science of implementation has offered little toward understanding how different implementation strategies work. To improve outcomes of implementation efforts, the field needs precise, testable theories that describe the causal pathways through which implementation strategies function. In this perspective piece, we describe a four-step approach to developing causal pathway models for implementation strategies.

Building causal models

First, it is important to ensure that implementation strategies are appropriately specified. Some strategies in published compilations are well defined but may not be specified in terms of its core component that can have a reliable and measureable impact. Second, linkages between strategies and mechanisms need to be generated. Existing compilations do not offer mechanisms by which strategies act, or the processes or events through which an implementation strategy operates to affect desired implementation outcomes. Third, it is critical to identify proximal and distal outcomes the strategy is theorized to impact, with the former being direct, measurable products of the strategy and the latter being one of eight implementation outcomes (1). Finally, articulating effect modifiers, like preconditions and moderators, allow for an understanding of where, when, and why strategies have an effect on outcomes of interest.

Future directions

We argue for greater precision in use of terms for factors implicated in implementation processes; development of guidelines for selecting research design and study plans that account for practical constructs and allow for the study of mechanisms; psychometrically strong and pragmatic measures of mechanisms; and more robust curation of evidence for knowledge transfer and use.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

✪ Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping

Authors:

Byron J. Powell, Cameo F. Stanick, Heather M. Halko, Caitlin N. Dorsey, Bryan J. Weiner, Melanie A. Barwick, Laura J. Damschroder, Michel Wensing, Luke Wolfenden, and Cara C. Lewis

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: October 2017

Read the full text in the open access journal Implementation Science

Abstract:

Background

Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria’s clarity and importance.

Methods

Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data.

Findings

The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented.

Conclusions

This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

Intentional research design in implementation science: Implications for the use of nomothetic and idiographic assessment

Authors:

Aaron R. Lyon, Elizabeth Connors, Amanda Jensen-Doss, Sara J. Landes, Cara C. Lewis, Bryce D. McLeod, Christopher Rutt, Cameo Stanick, Bryan J. Weiner

University of Washington affiliated authors are displayed in bold.

Subscription Required for Full Text Access

Published: September 2017

Read the full text in the subscription access journal Translational Behavioral Medicine

✪ Psychometric assessment of three newly developed implementation outcome measures

Authors:

Bryan J. Weiner, Cara C. Lewis, Cameo Stanick, Byron J. Powell, Caitlin N. Dorsey, Alecia S. Clary, Marcella H. Boynton and Heather Halko

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: August 2017

Read the full text in the open access journal Implementation Science

Abstract:

Background

Implementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).

Methods

Thirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change.

Results

All but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α’s from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions.

Conclusions

The AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**

Health Information Technologies — Academic and Commercial Evaluation (HIT-ACE) methodology: Description and application to clinical feedback systems

Authors:

Aaron R. Lyon, Cara C. Lewis, Abigail Melvin, Meredith Boyd, Semret Nicodimos, Freda F. Liu, Nathaniel Jungbluth

University of Washington affiliated authors are displayed in bold.

✪ Open Access

Published: September 2016

Read the full text in the open access journal Implementation Science

Abstract:

Background

Health information technologies (HIT) have become nearly ubiquitous in the contemporary healthcare landscape, but information about HIT development, functionality, and implementation readiness is frequently siloed. Theory-driven methods of compiling, evaluating, and integrating information from the academic and commercial sectors are necessary to guide stakeholder decision-making surrounding HIT adoption and to develop pragmatic HIT research agendas. This article presents the Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology, a structured, theory-driven method for compiling and evaluating information from multiple sectors. As an example demonstration of the methodology, we apply HIT-ACE to mental and behavioral health measurement feedback systems (MFS). MFS are a specific class of HIT that support the implementation of routine outcome monitoring, an evidence-based practice.

Results

HIT-ACE is guided by theories and frameworks related to user-centered design and implementation science. The methodology involves four phases: (1) coding academic and commercial materials, (2) developer/purveyor interviews, (3) linking putative implementation mechanisms to hit capabilities, and (4) experimental testing of capabilities and mechanisms. In the current demonstration, phase 1 included a systematic process to identify MFS in mental and behavioral health using academic literature and commercial websites. Using user-centered design, implementation science, and feedback frameworks, the HIT-ACE coding system was developed, piloted, and used to review each identified system for the presence of 38 capabilities and 18 additional characteristics via a consensus coding process. Bibliometic data were also collected to examine the representation of the systems in the scientific literature. As an example, results are presented for the application of HIT-ACE phase 1 to MFS wherein 49 separate MFS were identified, reflecting a diverse array of characteristics and capabilities.

Conclusions

Preliminary findings demonstrate the utility of HIT-ACE to represent the scope and diversity of a given class of HIT beyond what can be identified in the academic literature. Phase 2 data collection is expected to confirm and expand the information presented and phases 3 and 4 will provide more nuanced information about the impact of specific HIT capabilities. In all, HIT-ACE is expected to support adoption decisions and additional HIT development and implementation research.

**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**