Measure Development Archives - Implementation Science at UW

✪ “Going above and beyond” for implementation in the education sector: extension and validation of the School Implementation Citizenship Behavior Scale (SICBS)

Abstract:

Background

Employee behaviors that strategically support implementation (i.e., implementation citizenship behavior [ICB]) theoretically promote the adoption and high-fidelity use of evidence-based practices (EBPs). ICB (e.g., helping colleagues overcome implementation barriers) may vary across contexts, including schools where children are most likely to access and receive mental and behavioral health services. Pragmatic measures are needed to advance nascent research on school-based ICB and inform how these behaviors can be used to support successful implementation. The current study expanded the Implementation Citizenship Behavior Scale (ICBS) to create and validate the School Implementation Citizenship Behavior Scale (SICBS) in a sample of elementary school teachers implementing evidence-based prevention programs to support children’s mental and behavioral health.

Methods

Based on subject matter expert feedback, items were refined from the original ICBS and items for two new subscales (taking initiative, advocacy) were created for the SICBS. A sample of 441 public school teachers from 52 elementary schools in the Midwest and Western United States of America completed a survey that included the SICBS and additional measures to assess convergent and divergent validity. SICBS was refined and validated via examination of item characteristics curves to reduce items and develop a pragmatic instrument, confirmatory factor analyses to evaluate the hypothesized measurement structure, and assessment of convergent and divergent validity.

Results

The original two ICBS subscales (helping others, keeping informed) were retained, and two new three-item subscales resulted from item reduction analyses (taking initiative, advocacy). The hypothesized second-order factor model was generally well fit to the data (CFI = .99, TLI = .99, RMSEA = .09), all first- (λs = .85-.96) and second-order factor loadings (λs = .93-.95) were high. All SICBS subscales demonstrated acceptable reliability (αs = .88-.92). Convergent validity was evidenced by moderate correlations with organizational citizenship behavior items (rs = .42-.49). Divergent validity was demonstrated by weak correlations with teachers’ beliefs about teaching (rs = .31-.38) and null correlations with most school demographics.

Conclusion

Results support the structural, convergent, and divergent validity of the 12-item, 4-factor SICBS. The SICBS provides a deeper understanding of individual implementer actions that may serve as implementation mechanisms or outcomes.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers

Abstract:

Introduction

Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms.

Method

Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools–Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure.

Results

Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested.

Conclusion

The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale

Abstract:

Background

Usability—the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction—may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in this study of the IUS.

Method

A survey administered to 136 medical professionals from 11 primary-care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary-care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.

Results

Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.

Conclusion

This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Plain language abstract

The ease with which evidence-based psychosocial interventions (EBPIs) can be readily adopted and used by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability. No quantitative instruments exist to evaluate the usability of complex health interventions, such as the EBPIs that are commonly used to integrate mental and behavioral health services into primary care. This article describes the evaluation of the first quantitative instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS)

✪ Operationalizing the ‘pragmatic’ measures construct using a stakeholder feedback and a multi-method approach

Abstract:

Context

Implementation science measures are rarely used by stakeholders to inform and enhance clinical program change. Little is known about what makes implementation measures pragmatic (i.e., practical) for use in community settings; thus, the present study’s objective was to generate a clinical stakeholder-driven operationalization of a pragmatic measures construct.

Evidence acquisition

The pragmatic measures construct was defined using: 1) a systematic literature review to identify dimensions of the construct using PsycINFO and PubMed databases, and 2) interviews with an international stakeholder panel (N = 7) who were asked about their perspectives of pragmatic measures.

Evidence synthesis

Combined results from the systematic literature review and stakeholder interviews revealed a final list of 47 short statements (e.g., feasible, low cost, brief) describing pragmatic measures, which will allow for the development of a rigorous, stakeholder-driven conceptualization of the pragmatic measures construct.

Conclusions

Results revealed significant overlap between terms related to the pragmatic construct in the existing literature and stakeholder interviews. However, a number of terms were unique to each methodology. This underscores the importance of understanding stakeholder perspectives of criteria measuring the pragmatic construct. These results will be used to inform future phases of the project where stakeholders will determine the relative importance and clarity of each dimension of the pragmatic construct, as well as their priorities for the pragmatic dimensions. Taken together, these results will be incorporated into a pragmatic rating system for existing implementation science measures to support implementation science and practice.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Measuring Constructs of the Consolidated Framework for Implementation Research in the Context of Increasing Colorectal Cancer Screening in Federally Qualified Health Center

✪ Confirmatory factor analysis of the Evidence-Based Practice Attitudes Scale with school-based behavioral health consultants

Abstract:

Background

The Evidence-Based Practice Attitude Scale (EBPAS) is a widely used tool, but it has not been adapted and validated for use in schools, the most common setting where youth access behavioral health services. This study examined the factor structure, psychometric properties, and criterion-related validity of the school-adapted EBPAS in a sample of school-based behavioral health consultants.

Method

A research team comprised of experts in implementation of evidence-based practices in schools along with the original developer adapted the EBPAS for the school setting. The adapted instrument was administered to a representative sample (n = 196) of school-based behavioral health consultants to assess the reliability and structural validity via a series of confirmatory factor analyses.

Results

The original EBPAS factor structure was confirmed, with the final model supporting four first-order factors that load onto a second-order factor capturing general attitudes toward evidence-based practice. Correlations among the subscales indicated both unique and shared variance. Correlations between EBPAS scores and consultant variables demonstrated differential criterion-related validity, with the total score and the Requirements and Openness subscales demonstrating the strongest correlations.

Conclusions

The adapted EBPAS performed well when administered to behavioral health consultants operating in the educator sector, supporting the relevance of assessing attitudes in school settings. Potential directions for future research and applications of the EBPAS in schools and other service sectors are discussed.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Developing measures to assess constructs from the Inner Setting domain of the Consolidated Framework for Implementation Research

Abstract:

Background

Scientists and practitioners alike need reliable, valid measures of contextual factors that influence implementation. Yet, few existing measures demonstrate reliability or validity. To meet this need, we developed and assessed the psychometric properties of measures of several constructs within the Inner Setting domain of the Consolidated Framework for Implementation Research (CFIR).

Methods

We searched the literature for existing measures for the 7 Inner Setting domain constructs (Culture Overall, Culture Stress, Culture Effort, Implementation Climate, Learning Climate, Leadership Engagement, and Available Resources). We adapted items for the healthcare context, pilot-tested the adapted measures in 4 Federally Qualified Health Centers (FQHCs), and implemented the revised measures in 78 FQHCs in the 7 states (N = 327 respondents) with a focus on colorectal cancer (CRC) screening practices. To psychometrically assess our measures, we conducted confirmatory factor analysis models (CFA; structural validity), assessed inter-item consistency (reliability), computed scale correlations (discriminant validity), and calculated inter-rater reliability and agreement (organization-level construct reliability and validity).

Results

CFAs for most constructs exhibited good model fit (CFI > 0.90, TLI > 0.90, SRMR < 0.08, RMSEA < 0.08), with almost all factor loadings exceeding 0.40. Scale reliabilities ranged from good (0.7 ≤ α < 0.9) to excellent (α ≥ 0.9). Scale correlations fell below 0.90, indicating discriminant validity. Inter-rater reliability and agreement were sufficiently high to justify measuring constructs at the clinic-level.

Conclusions

Our findings provide psychometric evidence in support of the CFIR Inner Setting measures. Our findings also suggest the Inner Setting measures from individuals can be aggregated to represent the clinic-level. Measurement of the Inner Setting constructs can be useful in better understanding and predicting implementation in FQHCs and can be used to identify targets of strategies to accelerate and enhance implementation efforts in FQHCs.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping

Abstract:

Background

Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria’s clarity and importance.

Methods

Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data.

Findings

The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented.

Conclusions

This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Psychometric assessment of three newly developed implementation outcome measures

Abstract:

Background

Implementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).

Methods

Thirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change.

Results

All but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α’s from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions.

Conclusions

The AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Introduction

Method

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusion

Plain language abstract

Authors:

Subscription Required for Full Text Access

Read the full text in the subscription access journal Translational Behavioral Medicine

Authors:

Read the full text in the open access journal BMC Health Services Research

Abstract:

Context

Evidence acquisition

Evidence synthesis

Conclusions

Authors:

Read the full text in the subscription access journal Health Services Research

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Method

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Findings

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Results

Conclusions