Usability Archives - Page 3 of 4 - Implementation Science at UW

✪ A 4D Theoretical Framework for Measuring Topic-Specific Influence on Twitter: Development and Usability Study on Dietary Sodium Tweets

Abstract:

Background

Social media has emerged as a prominent approach for health education and promotion. However, it is challenging to understand how to best promote health-related information on social media platforms such as Twitter. Despite commercial tools and prior studies attempting to analyze influence, there is a gap to fill in developing a publicly accessible and consolidated framework to measure influence and analyze dissemination strategies.

Objective

We aimed to develop a theoretical framework to measure topic-specific user influence on Twitter and to examine its usability by analyzing dietary sodium tweets to support public health agencies in improving their dissemination strategies.

Methods

We designed a consolidated framework for measuring influence that can capture topic-specific tweeting behaviors. The core of the framework is a summary indicator of influence decomposable into 4 dimensions: activity, priority, originality, and popularity. These measures can be easily visualized and efficiently computed for any Twitter account without the need for private access. We demonstrated the proposed methods by using a case study on dietary sodium tweets with sampled stakeholders and then compared the framework with a traditional measure of influence.

Results

More than half a million dietary sodium tweets from 2006 to 2022 were retrieved for 16 US domestic and international stakeholders in 4 categories, that is, public agencies, academic institutions, professional associations, and experts. We discovered that World Health Organization, American Heart Association, Food and Agriculture Organization of the United Nations (UN-FAO), and World Action on Salt (WASH) were the top 4 sodium influencers in the sample. Each had different strengths and weaknesses in their dissemination strategies, and 2 stakeholders with similar overall influence, that is, UN-FAO and WASH, could have significantly different tweeting patterns. In addition, we identified exemplars in each dimension of influence. Regarding tweeting activity, a dedicated expert published more sodium tweets than any organization in the sample in the past 16 years. In terms of priority, WASH had more than half of its tweets dedicated to sodium. UN-FAO had both the highest proportion of original sodium tweets and posted the most popular sodium tweets among all sampled stakeholders. Regardless of excellence in 1 dimension, the 4 most influential stakeholders excelled in at least 2 out of 4 dimensions of influence.

Conclusions

Our findings demonstrate that our method not only aligned with a traditional measure of influence but also advanced influence analysis by analyzing the 4 dimensions that contribute to topic-specific influence. This consolidated framework provides quantifiable measures for public health entities to understand their bottleneck of influence and refine their social media campaign strategies. Our framework can be applied to improve the dissemination of other health topics as well as assist policy makers and public campaign experts to maximize population impact.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Co-developed implementation guidelines to maximize acceptability, feasibility, and usability of mobile phone supervision in Kenya

Abstract:

Opportunities exist to leverage mobile phones to replace or supplement in-person supervision of lay counselors. However, contextual variables, such as network connectivity and provider preferences, must be considered. Using an iterative and mixed methods approach, we co-developed implementation guidelines to support the implementation of mobile phone supervision with lay counselors and supervisors delivering a culturally adapted trauma-focused cognitive behavioral therapy in Western Kenya. Guidelines were shared and discussed with lay counselors in educational outreach visits led by supervisors. We evaluated the impact of guidelines and outreach on the acceptability, feasibility, and usability of mobile phone supervision. Guidelines were associated with significant improvements in acceptability and usability of mobile phone supervision. There was no evidence of a significant difference in feasibility. Qualitative interviews with lay counselors and supervisors contextualized how guidelines impacted acceptability and feasibility – by setting expectations for mobile phone supervision, emphasizing importance, increasing comfort, and sharing strategies to improve mobile phone supervision. Introducing and discussing co-developed implementation guidelines significantly improved the acceptability and usability of mobile phone supervision. This approach may provide a flexible and scalable model to address challenges with implementing evidence-based practices and implementation strategies in lower-resourced areas.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Time Utilization Among Immunization Clinics Using an Electronic Immunization Registry (Part 2): Time and Motion Study of Modified User Workflows

Abstract:

Background

Digital health interventions have the potential to improve the provision of health care services through digitized data collection and management. Low- and middle-income countries are beginning to introduce electronic immunization registries (EIRs) into their routine immunization services to better capture and store childhood vaccination information. Especially in Africa, where 25% of children remain unimmunized or underimmunized, technologies that can help identify children due for a vaccination are particularly important for improving vaccination coverage. However, an improved understanding of the effectiveness of these systems is needed to develop and deploy sustainable EIRs in low- and middle-income countries.

Objective

We conducted an interventional pretest-posttest design study that sought to improve time efficiency through workflow modifications in Kenyan immunization clinics. Our aim was to describe how activity times differed after introducing workflow modifications that could potentially reduce the time needed to perform routine data entry activities. Our intent was to demonstrate changes in efficiency when moving from the existing dual–data entry workflow to a future paperless workflow by health facility size and experience length of health care workers (HCWs).

Methods

We tested how 3 workflow modifications would affect time utilization among HCWs using the EIR at the point of care compared with baseline immunization clinic workflows. Our outcome of interest was the time taken to complete individual activities and a patient’s total time in the clinic where we compared the time spent during the baseline workflow with that during the modified workflow. We used a standardized tool to observe and document the immunization clinic workflow. To estimate differences in time utilization, we used bivariate analyses and fit multivariate linear mixed-effects models.

Results

Our study found that for HCWs using an EIR, the introduction of modified workflows decreased the amount of time needed to provide services to children seen in the immunization clinic. With a baseline mean time of 10 minutes spent per child, this decreased by about 3 minutes when the preparation modification was introduced and almost 5 minutes for the paperless and combined modifications. Results pertaining to the EIR’s performance and ability to connect to the internet were particularly insightful about potential causes of delays.

Conclusions

We were able to conduct a concise clinical simulation exercise by introducing modified workflows and estimating their impact on time utilization in immunization clinics using an EIR. We found that the paperless workflow provided the largest time savings when delivering services, although this was threatened by poor EIR performance and internet connectivity. This study demonstrated that not only should digital health interventions be built and adapted for particular use cases but existing user workflows also need to adapt to new technology.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Integration of a Digital Health Intervention Into Immunization Clinic Workflows in Kenya: Qualitative, Realist Evaluation of Technology Usability

Abstract:

Background

In an effort to increase vaccination coverage in low-resource settings, digital tools have been introduced to better track immunization records, improve data management practices, and provide improved access to vaccination coverage data for decision-making. Despite the potential of these electronic systems to improve the provision of health services, few digital health interventions have been institutionalized at scale in low- and middle-income countries.

Objective

In this paper, we aimed to describe how health care workers in Kenya had integrated an electronic immunization registry into their immunization clinic workflows and to use these findings to inform the development of a refined program theory on the registry’s usability.

Methods

Informed by realist methodology, we developed a program theory to explain usability of the electronic immunization registry. We designed a qualitative study based on our theory to describe the barriers and facilitators influencing data entry and use. Qualitative data were collected through semistructured interviews with users and workflow observations of immunization clinic sessions. Our findings were summarized by context-mechanism-outcome relationships formed after analyzing our key themes across interviews and workflow observations. Using these relationships, we were able to identify common rules for future implementers.

Results

Across the 12 facilities included in our study, 19 health care workers were interviewed, and 58 workflow sessions were observed. The common rules developed from our qualitative findings are as follows: rule 1—ensure that the users complete training to build familiarity with the system, understand the value of the system and data, and know where to find support; rule 2—confirm that the system captures all data needed for users to provide routine health care services and is easy to navigate; rule 3—identify work-arounds for poor network, system performance, and too few staff or resources; and rule 4—make users aware of expected changes to their workflow, and how these changes might differ over time and by facility size or number of patients. Upon study completion, we revised the program theory to reflect the importance of the goals and workflows of electronic immunization registries aligning with reality.

Conclusions

We created a deeper understanding of the underlying mechanisms for usability of the registry. We found that the electronic immunization registry had high acceptability among users; however, there were numerous barriers to using the system, even under ideal conditions, causing a misalignment between the system and the reality of the users’ workflows and their environment. Human-centered design and human-factors methods can assist during pilot stages to better align systems with users’ needs and again after scale-up to ensure that interventions are suitable for all user settings."

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ A sequential, multiple assignment randomized trial comparing web-based education to mobile video interpreter access for improving provider interpreter use in primary care clinics: the mVOCAL hybrid type 3 study protocol

Abstract:

Background

Individuals who use a language other than English for medical care are at risk for disparities related to healthcare safety, patient-centered care, and quality. Professional interpreter use decreases these disparities but remains underutilized, despite widespread access and legal mandates. In this study, we compare two discrete implementation strategies for improving interpreter use: (1) enhanced education targeting intrapersonal barriers to use delivered in a scalable format (interactive web-based educational modules) and (2) a strategy targeting system barriers to use in which mobile video interpreting is enabled on providers’ own mobile devices.

Methods

We will conduct a type 3 hybrid implementation-effectiveness study in 3–5 primary care organizations, using a sequential multiple assignment randomized trial (SMART) design. Our primary implementation outcome is interpreter use, calculated by matching clinic visits to interpreter invoices. Our secondary effectiveness outcome is patient comprehension, determined by comparing patient-reported to provider-documented visit diagnosis. Enrolled providers (n = 55) will be randomized to mobile video interpreting or educational modules, plus standard interpreter access. After 9 months, providers with high interpreter use will continue as assigned; those with lower use will be randomized to continue as before or add the alternative strategy. After another 9 months, both strategies will be available to enrolled providers for 9 more months. Providers will complete 2 surveys (beginning and end) and 3 in-depth interviews (beginning, middle, and end) to understand barriers to interpreter use, based on the Theoretical Domains Framework. Patients who use a language other than English will be surveyed (n = 648) and interviewed (n = 75) following visits with enrolled providers to understand their experiences with communication. Visits will be video recorded (n = 100) to assess fidelity to assigned strategies. We will explore strategy mechanism activation to refine causal pathway models using a quantitative plus qualitative approach. We will also determine the incremental cost-effectiveness of each implementation strategy from a healthcare organization perspective, using administrative and provider survey data.

Discussion

Determining how these two scalable strategies, alone and in sequence, perform for improving interpreter use, the mechanisms by which they do so, and at what cost, will provide critical insights for addressing a persistent cause of healthcare disparities.

Trial registration

NCT05591586

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Improving the Usability of Written Exposure Therapy for Therapists in the Department of Veterans Affairs Telemental Health: Formative Study Using Qualitative and User-Centered Design Methods

Abstract:

Background

User modifications are common in evidence-based psychosocial interventions (EBPIs) for mental health disorders. Often, EBPIs fit poorly into clinical workflows, require extensive resources, or pose considerable burden to patients and therapists. Implementation science is increasingly researching ways to improve the usability of EBPIs before implementation. A user-centered design can be used to support implementation methods to prioritize user needs and solutions to improve EBPI usability.

Objective

Trauma-focused EBPIs are a first-line treatment for patients with posttraumatic stress disorder (PTSD) in the Department of Veterans Affairs. Written exposure therapy (WET) is a brief, trauma-focused EBPI wherein patients handwrite about trauma associated with their PTSD. Initially developed for in-person delivery, WET is increasingly being delivered remotely, and outcomes appear to be equivalent to in-person delivery. However, there are logistical issues in delivering WET via video. In this evaluation, we explored usability issues related to WET telehealth delivery via videoconferencing software and designed a solution for therapist-facing challenges to systematize WET telehealth delivery.

Methods

In the Discover phase, both groups had challenges delivering WET and other PTSD therapies via telehealth because of technology issues with videoconferencing software, environmental distractions, and workflow disruptions. Narrative transfer (ie, patients sending handwritten trauma accounts to therapists) was the first target for design solution development as it was deemed most critical to WET delivery. In the Design and Build phase, we identified design constraints and brainstormed solution ideas. This led to the development of 3 solution workflows that were presented to a subgroup of therapist users through cognitive walkthroughs. Meetings with this subgroup allowed workflow refinement to improve narrative transfers. Finally, to facilitate using these workflows, we developed PDF manuals that are being refined in subsequent phases of the implementation project (not mentioned in this paper).

Results

Conclusion

The Discover, Design and Build, and Test framework can be a useful tool for understanding user needs in complex EBPI interventions and designing solutions to user-identified usability issues. Building on this work, an iterative evaluation of the 3 solution workflows and accompanying manuals with therapists and patients is underway as part of a nationwide WET implementation in telehealth settings.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Soil-transmitted helminth surveillance in Benin: A mixed-methods analysis of factors influencing non-participation in longitudinal surveillance activities

Abstract:

Background

Despite the significant success of deworming programs in reducing morbidity due to soil-transmitted helminth (STH) infections globally, efforts to achieve elimination of STH as a public health problem or to potentially interrupt transmission will require improving and intensifying surveillance. However, non-participation in surveillance threatens the ability of programs to adequately monitor program status and limited research has been conducted to investigate drivers of non-participation in stool-based surveillance.

Methodology/Principal findings

This mixed-methods exploratory sequential study took place in Comé, Benin in association with the DeWorm3 Project. Six focus group discussions were conducted with individuals invited to participate in annual DeWorm3 stool surveillance. Thematic analysis was used to identify facilitators and barriers to participation and inform the quantitative analysis. A mixed-effects logistic regression model was built using baseline DeWorm3 survey data to identify factors associated with non-participation. Qualitative and quantitative findings were merged for interpretation. Among the 7,039 individuals invited to participate in baseline stool surveillance, the refusal rate was 8.1%. Qualitative themes included: community members weighing community-level benefits against individual-level risks, circulating rumors about misuse of stool samples, interpersonal communication with field agents, and cultural norms around handling adult feces. The quantitative analysis demonstrated that adults were significantly less likely to provide a stool sample than school-aged children (OR:0.69, 95%CI: 0.55–0.88), a finding that converged with the qualitative results. Individuals from areas in the highest quartile of population density were more likely to refuse to participate (OR:1.71, 95%CI:1.16–2.52). Several variables linked to community-affinity aligned with qualitative results; residing mainly in the community (OR:0.36, 95%CI:0.20–0.66) and having lived in the community for more than 10 years (OR:0.82, 95%CI:0.54–1.25) decreased likelihood of refusal.

Conclusions/Significance

Optimizing STH surveillance will require that programs reimagine STH surveillance activities to address community concerns and ensure that no subpopulations are inadvertently excluded from surveillance data.

Author summary:

Soil-transmitted helminths (STH) are a group of intestinal parasites infecting approximately 1.5 billion people globally and resulting in significant adverse health outcomes. STH surveillance is conducted across endemic regions to assess prevalence of infection, to identify areas for mass drug administration implementation, and to monitor progress. The World Health Organization targets the elimination of STH as a public health problem in endemic settings with research currently being conducted to determine the feasibility of interrupting transmission of STH. In order to optimally design and manage programs towards these goals, and to verify whether elimination of STH as a public health problem has occurred, improvements in surveillance are needed. This mixed-methods study took place in Comé, Benin in association with the DeWorm3 Project, to identify drivers of non-participation in stool-based STH surveillance. This study found that certain individuals are more likely to refuse to participate in STH surveillance activities than others, including adults, individuals in urban areas, short-term residents in communities, and those perceiving their families to not be at risk for STH. As STH surveillance is intensified, programs will need to reimagine how surveillance is conducted to address community concerns and ensure that no subpopulations are inadvertently excluded from surveillance data.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability

Abstract:

Background

Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care.

Method

The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability.

Results

Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes.

Conclusions

CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale

Abstract:

Background

Usability—the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction—may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in this study of the IUS.

Method

A survey administered to 136 medical professionals from 11 primary-care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary-care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.

Results

Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.

Conclusion

This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Plain language abstract

The ease with which evidence-based psychosocial interventions (EBPIs) can be readily adopted and used by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability. No quantitative instruments exist to evaluate the usability of complex health interventions, such as the EBPIs that are commonly used to integrate mental and behavioral health services into primary care. This article describes the evaluation of the first quantitative instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

✪ Implementation Determinants and Outcomes of a Technology-Enabled Service Targeting Suicide Risk in High Schools: Mixed Methods Study

Abstract:

Background

Technology-enabled services (TESs), which integrate human service and digital components, are popular strategies to increase the reach and impact of mental health interventions, but large-scale implementation of TESs has lagged behind their potential.

Objective

This study applied a mixed qualitative and quantitative approach to gather input from multiple key user groups (students and educators) and to understand the factors that support successful implementation (implementation determinants) and implementation outcomes of a TES for universal screening, ongoing monitoring, and support for suicide risk management in the school setting.

Methods

A total of 111 students in the 9th to 12th grade completed measures regarding implementation outcomes (acceptability, feasibility, and appropriateness) via an open-ended survey. A total of 9 school personnel (school-based mental health clinicians, nurses, and administrators) completed laboratory-based usability testing of a dashboard tracking the suicide risk of students, quantitative measures, and qualitative interviews to understand key implementation outcomes and determinants. School personnel were presented with a series of scenarios and common tasks focused on the basic features and functions of the dashboard. Directed content analysis based on the Consolidated Framework for Implementation Research was used to extract multilevel determinants (ie, the barriers or facilitators at the levels of the outer setting, inner setting, individuals, intervention, and implementation process) related to positive implementation outcomes of the TES.

Results

Overarching themes related to implementation determinants and outcomes suggest that both student and school personnel users view TESs for suicide prevention as moderately feasible and acceptable based on the Acceptability of Intervention Measure and Feasibility of Intervention Measure and as needing improvements in usability based on the System Usability Scale. Qualitative results suggest that students and school personnel view passive data collection based on social media data as a relative advantage to the current system; however, the findings indicate that the TES and the school setting need to address issues of privacy, integration into existing workflows and communication patterns, and options for individualization for student-centered care.

Conclusions

Innovative suicide prevention strategies that rely on passive data collection in the school context are a promising and appealing idea. Usability testing identified key issues for revision to facilitate widespread implementation.

^{**This abstract is posted with permission under the Creative Commons Attribution 4.0 International License**}

Authors:

✪ Open Access

Read the full text in the open access journal Journal of Medical Internet Research

Abstract:

Background

Objective

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Cambridge Prisms: Global Mental Health

Abstract:

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Formative Research

Abstract:

Background

Objective

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Formative Research

Abstract:

Background

Objective

Methods

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science

Abstract:

Background

Methods

Discussion

Trial registration

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Formative Research

Abstract:

Background

Objective

Methods

Results

Conclusion

Authors:

✪ Open Access

Read the full text in the open access journal PLOS Neglected Tropical Diseases

Abstract:

Background

Methodology/Principal findings

Conclusions/Significance

Author summary:

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Science Communications

Abstract:

Background

Method

Results

Conclusions

Authors:

✪ Open Access

Read the full text in the open access journal Implementation Research and Practice

Abstract:

Background

Method

Results

Conclusion

Plain language abstract

Authors:

✪ Open Access

Read the full text in the open access journal JMIR Mental Health

Abstract:

Background

Objective

Methods