Overview of Study Designs in Implementation Science
Implementation science seeks to improve the adoption, adaptation, delivery and sustainment of evidence-based interventions in healthcare, and central to this goal is understanding how interventions are delivered effectively in the context of the 7 P’s.
Research designed to evaluate the impact of these contexts takes many forms, and design selection is critical to capturing data in a manner that appropriately addresses your research question or questions.
Implementation research largely attends to external validity, whereas most randomized efficacy and effectiveness research designs emphasize internal validity.
Given these differing focal points, a debate exists in the field as to the role of randomized design in implementation research and the relative merit of quantitative, qualitative, and mixed methods designs.
Open Access articles will be marked with ✪
Please note some journals will require subscriptions to access a linked article.
Learn More:
To learn more about which study designs are used in implementation research and why, please see below. In addition to two helpful overview articles about research design in implementation research, the National Cancer Institute’s archived Fireside Chat on study designs is a useful primer.
- ✪ An Overview of Research and Evaluation Designs for Dissemination and Implementation (Annual Review of Public Health, 2017)
- ✪ Variation in Research Designs Used to Test the Effectiveness of Dissemination and Implementation Strategies: A Review (Frontiers in Public Health, 2018)
- ✪ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review (Implementation Science, 2019)
Selected Study Designs Used in Implementation Science
Randomized Control Trials
Randomized Control Trial
A type of experimental clinical study where participants are randomly selected into either the intervention or control group and then followed over time.
Learn More:
Experimental and quasi-experimental designs in implementation research (Psychiatry Research, 2019)
Designing and undertaking randomised implementation trials: Guide for researchers (BMJ, 2021)
Cluster Randomized Control Trial
A randomized control trial with parallel or cross-over design that randomizes pre-existing groups into intervention and control groups.
Methodology
- ✪ How to design efficient cluster randomised trials (BMJ, 2017)
- ✪ Trials and tribulations: cross-learning from the practices of epidemiologists and economists in the evaluation of public health interventions (Health Policy and Planning, 2018)
Examples of Use
- ✪ Impact of a tailored program on the implementation of evidence-based recommendations for multimorbid patients with polypharmacy in primary care practices—results of a cluster-randomized controlled trial (Implementation Science, 2017)
- ✪ Cluster randomised controlled trial of a theory-based multiple behaviour change intervention aimed at healthcare professionals to improve their management of type 2 diabetes in primary care (Implementation Science, 2018)
- ✪ Systems analysis and improvement approach to optimize the hypertension diagnosis and care cascade for PLHIV individuals (SAIA-HTN): a hybrid type III cluster randomized trial (Implementation Science, 2020)
Stepped-Wedge Design
A randomized cross-over design where different clusters cross into the intervention condition at different points in time, all clusters receive the intervention, and the intervention state continues in a given cluster once begun.
Methodology
- ✪ Stepped‐wedge cluster randomised controlled trials: a generic framework including parallel and multiple‐level designs (Statistics in Medicine, 2014)
- Research Designs for Intervention Research with Small Samples II: Stepped Wedge and Interrupted Time-Series Designs (Prevention Science, 2015)
- ✪ The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting (The BMJ, 2015)
- ✪ Designing a stepped wedge trial: Three main designs, carry-over effects and randomisation approaches (Trials, 2015)
- ✪ Five questions to consider before conducting a stepped wedge trial (Trials, 2015)
- ✪ Analysis and reporting of stepped wedge randomised controlled trials: synthesis and critical appraisal of published studies, 2010 to 2014 (Trials, 2015)
- Evaluating Public Health Interventions: 2. Stepping Up to Routine Public Health Evaluation With the Stepped Wedge Design (American Journal of Public Health, 2016)
Examples of Use
- ✪ Early ART initiation among HIV-positive pregnant women in central Mozambique: a stepped wedge randomized controlled trial of an optimized Option B+ approach (Implementation Science, 2015)
- An implementation science protocol of the Women’s Health CoOp in healthcare settings in Cape Town, South Africa: A stepped-wedge design (BMC Women's Health, 2017)
Definition: A design that focuses both on assessing clinical effectiveness and on implementation.
Curran et al (2012)1 proposed three types of effectiveness-implementation hybrid designs:
- Testing effects of a clinical intervention on relevant outcomes while observing and gathering information on implementation
- Dual testing of clinical and implementation interventions/strategies
- Testing of an implementation strategy while observing and gathering information on the clinical intervention's impact on relevant outcomes
💻 “Hybrid Designs” Combining Elements of Clinical Effectiveness and Implementation Research
Methodology
- 1Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact (Medical Care, 2012)
- ✪ Effectiveness-implementation hybrid designs: implications for quality improvement science (Implementation Science, 2013)
- An introduction to effectiveness-implementation hybrid designs (Psychiatry Research, 2019)
- ✪ Expanding Hybrid Studies for Implementation Research: Intervention, Implementation Strategy, and Context (Frontiers in Public Health, 2019)
Examples of Use
- ✪ A hybrid effectiveness-implementation cluster randomized trial of group CBT for anxiety in urban schools: rationale, design, and methods (Implementation Science, 2016)
- ✪ A Hybrid III stepped wedge cluster randomized trial testing an implementation strategy to facilitate the use of an evidence-based practice in VA Homeless Primary Care Treatment Programs (Implementation Science, 2017)
- ✪ Implementation findings from a hybrid III implementation-effectiveness trial of the Diabetes Prevention Program (DPP) in the Veterans Health Administration (VHA) (Implementation Science, 2017)
- ✪ Using a continuum of hybrid effectiveness-implementation studies to put research-tested colorectal screening interventions into practice (Implementation Science, 2019)
In 2009, Thorpe and colleagues published A PRagmatic–Explanatory Continuum Indicator Summary (PRECIS): A tool to help trial designers (CMAJ, 2009) to help researchers ensure that decisions made about trial design are aligned with the trial's stated purpose. This tool was refined in 2013 by Loudon and colleagues in ✪ PRECIS-2: A tool to improve the applicability of randomised controlled trials (Trials, 2013).
The PRECIS-2 tool organizes the nine elements for consideration into a wheel, consisting of:
- Eligibility criteria
- Setting
- Organization
- Flexibility of delivery
- Flexibility of adherence
- Follow up
- Primary outcome
- Primary analysis
To access the PRECIS-2 tool and to learn more about it, please visit PRECIS-2.org.
Quasi-Experimental Designs
Quasi-Experimental Designs
This category encompasses a broad range of nonrandomized intervention studies, frequently used when not logistically or ethically feasible to conduct a randomized control trial.1
Methodology
- ✪ Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research (Annual Review of Public Health, 2018)
- Big-5 Quasi-Experimental designs (Journal of Clinical Epidemiology, 2017)
- Quasi-experimental study designs series — Paper 2: Complementary approaches to advancing global health knowledge (Journal of Clinical Epidemiology, 2017)
Interrupted Time Series
Where routine monitoring data is collected at evenly-spaced time points before and after the intervention and the data from prior to intervention serves as the control group.
Methodology
- A robust interrupted time series model for analyzing complex health care intervention data (Statistics in Medicine, 2017)
- A matching framework to improve causal inference in interrupted time‐series analysis (Journal of Evaluation in Clinical Practice, 2017)
- Heterogeneity in application, design, and analysis characteristics was found for controlled before-after and interrupted time series studies included in Cochrane reviews (Journal of Clinical Epidemiology, 2017)
Examples of Use
- ✪ The use of segmented regression in analysing interrupted time series studies: an example in pre-hospital ambulance care (Implementation Science, 2014)
- ✪ Bridging the Gap: using an interrupted time series design to evaluate systems reform addressing refugee maternal and child health inequalities (Implementation Science, 2015)
- The antibiotic management of gonorrhoea in Ontario, Canada following multiple changes in guidelines: an interrupted time-series analysis (Sexually Transmitted Infections, 2017)
- ✪ Effect of a population-level performance dashboard intervention on maternal-newborn outcomes: an interrupted time series study (BMJ Quality & Safety, 2017)
Regression Discontinuity
A quasi-experimental design using pretest-posttest examination of causal effects of interventions, where a threshold determines which recipients receive the intervention. This allows for the estimation of average treatment effect when randomization is not feasible.
Methodology
- Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: a review of current and best practice (Journal of Clinical Epidemiology, 2015)
- Advancing Quality Improvement with Regression Discontinuity Designs (Annals of the American Thoracic Society, 2018)
- Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs (Journal of Business & Economic Statistics, 2017)
- Regression discontinuity designs in healthcare research (BMJ, 2016)
- ✪ Regression Discontinuity Designs in Epidemiology: Causal Inference Without Randomized Trials (Epidemiology, 2014)
- Alternatives to Randomized Control Trial Designs for Community-Based Prevention Evaluation (Prevention Science, 2017)
Examples of Use
- Effect of human papillomavirus (HPV) vaccination on clinical indicators of sexual behaviour among adolescent girls: the Ontario Grade 8 HPV Vaccine Cohort Study (Canadian Medical Association Journal, 2015)
- Challenges to Using the Regression Discontinuity Design in Educational Evaluations: Lessons From the Transition to Algebra Study (American Journal of Evaluation, 2016)
- School turnaround in North Carolina: A regression discontinuity analysis (Economics of Education Review, 2017)
Regression Point Displacement
A pre-test post-test quasi-experimental design where data is collected at the group level and typically involves one treatment group and more than one control group.
Methodology
- Evaluating Program Effectiveness Using the Regression Point Displacement Design (Evaluation & the Health Professions, 2006)
- Alternatives to Randomized Control Trials: A Review of Three Quasi-experimental Designs for Causal Inference (Actualidades en Psicología, 2015)
- Designs for Testing Group-Based Interventions with Limited Numbers of Social Units: The Dynamic Wait-Listed and Regression Point Displacement Designs (Prevention Science, 2015)
Intervention Optimization
Multiphase Optimization Strategy
A strategy for building, optimizing, and evaluating interventions using a three-phase method to identify active intervention components and which levels of each component lead to optimal outcomes.
SMART Design
A randomized experimental design developed particularly for building time-varying adaptive interventions.
Methodology
- The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART) (The American Journal of Preventive Medicine, 2007)
- ✪ Achieving the Goals of Translational Science in Public Health Intervention Research: The Multiphase Optimization Strategy (MOST) ( American Journal of Public Health, 2019)
- ✪ Randomization Procedures for Multicomponent Behavioral Intervention Factorial Trials in the Multiphase Optimization Strategy Framework: Challenges and Recommendations (Translational Behavioral Medicine, 2019)
Examples of Use
- Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST) (Translational Behavioral Medicine, 2014)
- ✪ Tobacco dependence treatment in the emergency department: A randomized trial using the Multiphase Optimization Strategy (Contemporary Clinical Trials, 2018)
Methodology
- The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART) (The American Journal of Preventive Medicine, 2007)
- Sequential Multiple-Assignment Randomized Trials: Developing and Evaluating Adaptive Interventions in Special Education ( Remedial and Special Education, 2018)
- SMART Thinking: a Review of Recent Developments in Sequential Multiple Assignment Randomized Trials (Current Epidemiology Reports, 2016)
Examples of Use
- ✪ High-Yield HIV Testing, Facilitated Linkage to Care, and Prevention for Female Youth in Kenya (GIRLS Study): Implementation Science Protocol for a Priority Population (JMIR Research Protocols, 2017)
- Getting “SMART” about implementing multi-tiered systems of support to promote school mental health (Journal of School Psychology, 2018)
- Sequential multiple assignment randomization trial designs for nursing research (Research in Nursing & Health, 2019)
Mixed Methods
Mixed Methods
The combination of at least one numerical (quantitative) research method and at least one non-numerical (qualitative) research method into a single study design.
💻 Advanced Topics for Implementation Science Research: Mixed Methods in Implementation Science
Methodology
- ✪ Mixed Method Designs in Implementation Research (Administration and Policy in Mental Health and Mental Health Services Research, 2011)
- Optimizing Mixed Methods for Implementation Research in Large Systems (Administration and Policy in Mental Health and Mental Health Services Research, 2015)
- Qualitative and Mixed Methods Research in Dissemination and Implementation Science (Journal of Clinical Child & Adolescent Psychology, 2014)
- ✪ Combining the Power of Stories and the Power of Numbers: Mixed Methods Research and Mixed Studies Reviews (Annual Review of Public Health, 2013)
- Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research (Administration and Policy in Mental Health and Mental Health Services Research, 2015)
Examples of Use
- ✪ Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation (Implementation Science, 2015)
- ✪ Lessons learnt during the process of setup and implementation of the voucher scheme in Eastern Uganda: a mixed methods study (Implementation Science, 2015)
- ✪ Understanding the impact of external context on community-based implementation of an evidence-based HIV risk reduction intervention (BMC Health Services Research, 2018)
- ✪ School health implementation tools: a mixed methods evaluation of factors influencing their use (Implementation Science, 2018)
Concept Mapping
A mixed-methods (qualitative and quantitative) approach for data collection and analysis that incorporates input from all participants in order to identify dimensions of productive collaboration and assess their importance and feasibility.1