Education Drivers

Effective Instruction Treatment Integrity

For the best chance of a positive impact on educational outcomes, two conditions must be met: (a) Effective interventions must be adopted, and (b) those interventions must be executed with sufficient quality (treatment integrity) to ensure the intervention is implemented as designed. Treatment integrity research finds that integrity scores are very low and, in practice, implementation is rarely assessed. If an intervention with a strong research base is not implemented with a high level of treatment integrity, then the students do not actually experience the intervention and there is no reason to assume they will benefit from it. Under these circumstances, it is impossible to know if poor outcomes are the result of an ineffective intervention or poor implementation of that intervention. Treatment integrity is defined as consistently implementing the essential components of an intervention as planned, intended, and originally designed. Studies find degradation of fidelity begins shortly after training. To ensure consistent levels of treatment integrity, teachers require ongoing monitoring and feedback of implementation. To date, research has not identified what threshold of treatment integrity is required to ensure the optimum benefit of an evidence-based practice.

 

Effective Instruction Treatment Integrity

Treatment Integrity PDF

Citation: Detrich, R., States, J., & Keyworth, R. (2017). Effective Instruction Treatment Integrity.Oakland, CA: The Wing Institute. https://www.winginstitute.org/effective-instruction-treatment-integrity.

For the best chance of producing positive educational outcomes for all children, two conditions must be met: (a) adopting effective empirically supported (evidence-based) practices and (b) implementing those practices with sufficient quality that they make a difference (treatment integrity) (Detrich, 2014). Both are necessary as neither on its own is sufficient to result in positive outcomes. Figure 1 describes the relationship between empirically supported practices and treatment integrity.

        

Treatment Int Fig 1 

Figure 1. Relationship between empirically supported practices and treatment integrity

 

            If an intervention has strong empirical support and is implemented with high integrity, then there is a high probability that positive outcomes will be achieved (upper left quadrant). The other quadrants illustrate that the lack of either element reduces the probability of positive outcomes: If a well-supported, research-based intervention is not implemented with high integrity (lower left quadrant); if an intervention is implemented with high integrity but is not empirically supported (upper right quadrant); or if a nonempirically supported intervention is implemented poorly (lower right quadrant).

            It should be noted that some interventions are not empirically supported because sufficient research has demonstrated that they do not produce positive outcomes. Alternatively, some interventions are not empirically supported because they have not been experimentally evaluated. This latter class of interventions is still in the experimental phase of development. Using these interventions is tantamount to conducting research and all of the rules for engaging in research should be followed. Because these interventions are still experimental, their effectiveness is unknown and they should be implemented only with the fully informed consent of both educators and parents.

            The advent of the evidence-based practice movement in education has resulted in considerable effort to identify practices that are empirically supported. Organizations such as the What Works Clearinghouse (https://ies.ed.gov/ncee/wwc) and the Best Evidence Encyclopedia (http://www.bestevidence.org) have reviewed a large number of interventions to discern the research base for each intervention. Until the past 20 years, treatment integrity did not receive significant scholarly attention. Even with the increased attention, treatment integrity measures are reported in about half of the published intervention research reports (Sanetti, Dobey, & Gritter, 2012; Sanetti, Gritter, & Dobey, 2011). When treatment integrity data are not published in research reports, it is difficult to know if the intervention that the researchers reported was actually implemented and was responsible for the outcomes or if there was some undocumented variation of the intervention that actually accounted for the outcomes.

           As important as treatment integrity is in research, it is equally important in practice settings. Measures of treatment integrity are fundamental to data-based decision making. Effective interventions may be prematurely terminated if the level of treatment integrity is not known. Student performance data tell us only how well a student is responding to the intervention as it is implemented. Treatment integrity measures tell us how well the intervention is being implemented. Without knowing about treatment integrity, it is not possible to know if a student’s failure to benefit from an intervention is a function of an ineffective intervention or ineffective implementation (see Treatment Integrity in the Problem-Solving Process for more discussion).

References

Detrich, R. (2014). Treatment integrity: Fundamental to education reform. Journal of Cognitive Education and Psychology, 13(2), 258–271.

Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14(1), 29–46.

Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40(1), 72–84.

Publications

TITLE
SYNOPSIS
CITATION
Treatment Integrity: Fundamental to Education Reform

To produce better outcomes for students two things are necessary: (1) effective, scientifically supported interventions (2) those interventions implemented with high integrity.  Typically, much greater attention has been given to identifying effective practices.  This review focuses on features of high quality implementation.

Detrich, R. (2014). Treatment integrity: Fundamental to education reform. Journal of Cognitive Education and Psychology, 13(2), 258-271.

Innovation, Implementation Science, and Data-Based Decision Making: Components of Successful Reform

Schools are often expected to implement innovative instructional programs.  Most often these initiatives fail because what we know from implementation science is not considered as part of implementing the initiative.  This chapter reviews the contributions implementation science can make for improving outcomes for students.

Detrich, R. Innovation, Implementation Science, and Data-Based Decision Making: Components of Successful Reform. Handbook on Innovations in Learning, 31.

Treatment Integrity: A Fundamental Unit of Sustainable Educational Programs.

Reform efforts tend to come and go very quickly in education. This paper makes the argument that the sustainability of programs is closely related to how well those programs are implemented.

Detrich, R., Keyworth, R. & States, J. (2010). Treatment Integrity: A Fundamental Unit of Sustainable Educational Programs. Journal of Evidence-Based Practices for Schools, 11(1), 4-29.

Approaches to Increasing Treatment Integrity

Strategies designed to increase treatment integrity fall into two categories: antecedent-based strategies and consequence-based strategies.

Detrich, R., States, J. & Keyworth, R. (2017). Approaches to Increasing Treatment Integrity. Oakland, Ca. The Wing Institute

 

Dimensions of Treatment Integrity Overview

Historically, treatment integrity has been defined as implementation of an intervention as planned (Gresham, 1989). More recently, treatment integrity has been reimagined as multidimensional (Dane & Schneider, 1998). In this conceptualization of treatment integrity are four dimensions relevant to practice: (a) exposure (dosage), (b) adherence, (c) quality of delivery, and (d) student responsiveness.  It is important to understand that these dimensions do not stand alone but rather interact to impact the ultimate effectiveness of an intervention. It is important for educators to assess all dimensions of treatment integrity to assure that it is being implemented as intended.

Detrich, R., States, J. & Keyworth, R. (2017). Dimensions of Treatment Integrity Overview. Oakland, Ca. The Wing Institute

 

Treatment Integrity in the Problem Solving Process

The usual approach to determining if an intervention is effective for a student is to review student outcome data; however, this is only part of the task. Student data can only be understood if we know something about how well the intervention was implemented. Student data without treatment integrity data are largely meaningless because without knowing how well an intervention has been implemented, no judgments can be made about the effectiveness of the intervention. Poor outcomes can be a function of an ineffective intervention or poor implementation of the intervention. Without treatment integrity data, the is a risk that an intervention will be judged as ineffective when, in fact, the quality of implementation was so inadequate that it would be unreasonable to expect positive outcomes.

Detrich, R., States, J. & Keyworth, R. (2017). Treatment Integrity in the Problem Solving Process. Oakland, Ca. The Wing Institute.

 

Overview of Treatment Integrity

For the best chance of a positive impact on educational outcomes, two conditions must be met: (a) Effective interventions must be adopted, and (b) those interventions must be implemented with sufficient quality (treatment integrity) to ensure benefit.  To date, emphasis in education has been on identifying effective interventions and less concern with implementing the interventions. The research on the implementation of interventions is not encouraging. Often, treatment integrity scores are very low and, in practice, implementation is rarely assessed. If an intervention with a strong research base is not implemented with a high level of treatment integrity, then the students do not actually experience the intervention and there is no reason to assume they will benefit from it. Under these circumstances, it is not possible to know if poor outcomes are the result of an ineffective intervention or poor implementation of that intervention. Historically, treatment integrity has been defined as implementing an intervention as prescribed. More recently, it has been conceptualized as having multiple dimensions, among them dosage and adherence which must be measured to ensure that it is occurring at adequate levels.

Detrich, R., States, J., & Keyworth, R. (2107). Overview of Treatment Integrity. Oakland, Ca. The Wing Institute.

Using Performance Feedback to Improve Teacher Effectiveness

This paper examines intervention and instruction failures and describe concrete steps that implementers can take to improve the results of their instruction and intervention in classrooms.

VanDerHeyden, A. (2013). Are We Making the Differences That Matter in Education? In Performance Feedback: Using Data to Improve Educator Performance (Vol. 3, pp. 119-138). Oakland, CA: The Wing Institute.

 

Presentations

TITLE
SYNOPSIS
CITATION
Using Performance Feedback to Improve Teacher Effectiveness

This paper examines intervention and instruction failures and describe concrete steps that implementers can take to improve the results of their instruction and intervention in classrooms.

VanDerHeyden, A. (2011). Using Performance Feedback to Improve Teacher Effectiveness [Powerpoint Slides]. Retrieved from 2011-wing-presentation-amanda-vanderheyden.

Treatment Integrity and Program Fidelity: Necessary but Not Sufficient to Sustain Programs
If programs are to sustain they must be implemented with integrity. If there is drift over time, it raises questions about whether the program is sustaining or has been substantially changed.
Detrich, R. (2008). Treatment Integrity and Program Fidelity: Necessary but Not Sufficient to Sustain Programs [Powerpoint Slides]. Retrieved from 2008-aba-presentation-ronnie-detrich.
Treatment Integrity: A Fundamental Component of PBS
School-wide initatives have to be well implemented if there is to be any benefit. This talk describes methods for assuring high levels of treatment integrity.
Detrich, R. (2008). Treatment Integrity: A Fundamental Component of PBS [Powerpoint Slides]. Retrieved from 2008-apbs-txint-presentation-ronnie-detrich.
Toward a Technology of Treatment Integrity
If research supported interventions are to be effective it is necessary that they are implemented with integrity. This paper describes approahes to assuring high levels of treatment integrtiy.
Detrich, R. (2011). Toward a Technology of Treatment Integrity [Powerpoint Slides]. Retrieved from 2011-apbs-presentation-ronnie-detrich.
Treatment Integrity: Necessary by Not Sufficient for Improving Outcomes
Treatment integrity is necessary to improve outcomes but it is not sufficient. It is also necessary to implement scientifically supported interventions.
Detrich, R. (2015). Treatment Integrity: Necessary by Not Sufficient for Improving Outcomes [Powerpoint Slides]. Retrieved from 2015-ebpindisabilities-txint-presentation-ronnie-detrich.

 

Student Research

TITLE
SYNOPSIS
CITATION
Preparing teachers to train parents in the use of evidence-based strategies for reading fluency.
This study evaluated the effects of training teachers to train parents to implement a program for increasing reading fluency for struggling readers. While all students improved, the quality of implementation moderated effects.
Kupzyk, S. (2009). Preparing teachers to train parents in the use of evidence-based strategies for reading fluency. Retrieved from student-research-2009-c.
TITLE
SYNOPSIS
CITATION
A Meta-Analysis of Direct Instruction

A soon to be published meta-analysis of Direct Instruction (DI) curricula that reviews research on DI curricula between 1966-2016 reports that DI curricula produced moderate to large effect sizes across the curriculum areas reading, math, language, and spelling.  The review is notable because it reviews a much larger body of DI research than has occurred in the past and covers a wide range of experimental designs (from single subject to randomized trials).  328 studies were reviewed and almost 4,000 effects were considered.  Given the variability in research designs and the breadth of the effects considered, it suggests that DI curricula produce robust results.  There was very little decline during maintenance phases of the study and greater exposure to the curricula resulted in greater effects.

Stockard, J., Wood, T. W., Coughlin, C. & Khoury, C. R. (in press), Review of Educational Research.  DOI: 10.3102/0034654317751919

 

TITLE
SYNOPSIS
Association for Behavior Analysis International (ABAI)
ABAI organization promotes the development, and support of the science of behavior analysis through research, education, and practice.
Center on Teaching and Learning (CTL)
CTL is research center that conducts and disseminates research that focuses on practical solutions to serious problems in school systems.
National Center on Student Progress Monitoring
The Center's mission is to provide technical assistance to states and districts and disseminate information about progress monitoring practices proven to work in different academic content areas (Gr. K-5).
Back to Top