Education Drivers

Problem Solving

Treatment Integrity in the Problem-Solving Process Overview

View this Overview as PDF

Treatment integrity is a core component of data-based decision making (Detrich, 2013). The usual approach is to consider student data when making decisions about an intervention; however, if there are no data about how well the intervention was implemented, then meaningful judgments cannot be made about effectiveness. Figure 1 contains hypothetical data to illustrate the necessity of treatment integrity data when making decisions about interventions. In this example, a student was identified as

ProblemSolvingFig1 

Figure 1 The analytical challenge of understanding the gap between expectations and actual performance

 

struggling with reading. Before the intervention, the student was reading grade-level materials at about 40 words correct per minute. The grade-level norm is about 120 words correct per minute. To close the achievement gap, an increase of two words correct per minute per week was established as a goal. After several weeks of intervention, the student’s actual improvement was only one word correct per minute per week. Although that slight increase constitutes progress, it will leave the student far behind his or her peers at the end of the year.

            The analytical task for the educator is to determine whether the intervention is sufficiently effective to achieve adequate growth. This cannot be determined unless there are data about how well the intervention was implemented. If treatment integrity data indicate that the intervention was implemented with high quality, then an alternative intervention should be considered. On the other hand, if the data indicate that the intervention was poorly implemented, then efforts should be taken to improve the quality of implementation before making any decisions about the effectiveness of the intervention.

            To make decisions about an intervention without treatment integrity data increases the risk that effective interventions will be discontinued because of apparent lack of progress. Although significant resources are required to assess treatment integrity, they are also required to implement an intervention poorly and then terminate it because of the incorrect assumption of failure.

A Pragmatic Problem-Solving Approach to Assessing Treatment Integrity

Measuring treatment integrity requires time and resources, both of which are in short supply in most schools. If assessing treatment integrity is an integral part of data-based decision making, then effective and efficient methods for doing so must be developed. One approach to increasing efficiency is illustrated in Figure 2.

 ProblemSolvingFig2

Figure 2. A pragmatic approach to assessing and intervening to improve treatment integrity

            Used with permission. Originally published in Detrich, R. (2013). Innovation, implementation   science, and data-based decision making: Components of successful reform. In M. Murphy, S.   Redding, & J. Twyman (Eds.), Handbook on Innovations in Learning (pp. 33–49). Charlotte, NC:          Information Age Publishing.

 

            This approach is based on the notion that because it is difficult to measure all of the dimensions of treatment integrity at the same time, a better alternative is to measure them sequentially. A pragmatic choice is made to assess treatment integrity when a student is not benefitting from instruction. Perhaps the easiest dimension of treatment integrity to assess is exposure (dosage). If exposure is not occurring as prescribed, then efforts should be made to improve exposure before assessing any other dimensions. To do this, it is necessary to determine if there are organizational variables or other factors that make it difficult to deliver instruction at the frequency and duration defined in the intervention protocol. The problem may be insufficient time in the school schedule for the intervention to occur as planned. Lack of time can impact both frequency and duration of exposure. Once exposure has been improved and the student is receiving instruction as prescribed, student data can be monitored to determine if performance has improved. If it has improved to adequate levels, then no further assessment is required.

            On the other hand, if student performance remains below expectations even though exposure is occurring as specified, it will be necessary to assess the next aspect of treatment integrity. Adherence should be assessed next. Research suggests that high adherence has a greater impact on student outcomes than any other dimension of integrity (Sanetti & Fallon, 2011); however, the different dimensions interact with each other and the combined effects can strengthen or weaken student outcomes.

            If adherence is adequate, then assess quality of delivery. However, if assessed adherence is less than optimal, then the next step is to improve adherence. Poor adherence may result when those responsible for implementation do not have the skills to carry out the intervention (can’t do) or are not motivated to implement it because they do not perceive the intervention to be a solution to the identified problem (won’t do). A third possibility is that the intervention is not a good contextual fit with the classroom culture and routines. Once the problem has been identified and action taken to improve adherence, student data should again be reviewed to determine if performance has improved to acceptable levels. If it has improved sufficiently, there is no need to assess other dimensions of treatment integrity.

            If student performance is still not at acceptable levels, then the next step is to assess quality of delivery. Both exposure and adherence can be objectively measured by direct observation. Assessing quality of delivery is more difficult because it is a more subjective measure. Consider the challenge of measuring praise. Certain aspects can be objectively measured; for example, specific praise can be discriminated from general praise. The qualitative features of praise are much more difficult to measure objectively. Measuring enthusiasm and sincerity is challenging but it requires assessment as it can impact student performance. Delivering praise in a monotone or robotically may result in poor student outcomes. Saying the same words with sincerity and enthusiasm is likely to result in very different outcomes.

            If quality of delivery is inadequate, then action should be taken to improve it before making decisions about the effectiveness of the intervention on student performance. As with the previous steps, once the assessed quality of delivery is at an adequate level, data should be reviewed to determine if the student is making adequate progress. If the student is making adequate progress, then nothing further needs to be done.

            If the student is not making adequate progress, then the final dimension of treatment integrity, student responsiveness, should be assessed. If student responsiveness, like all the other dimensions of treatment integrity, is sufficiently high, then a new intervention should be considered. If student engagement is low, then efforts should be taken to increase it before making any decisions about the intervention.  

            In summary, if a particular dimension of treatment integrity is increased to higher levels and student performance data indicate that adequate progress is being made then no further assessment of the other dimensions of treatment integrity is required. On the other hand, if student performance does not improve when a particular dimension of treatment integrity is increased then it is necessary to keep assessing and improving the other dimensions of treatment integrity until the student is making adequate progress. If all dimensions of treatment integrity are being implemented at high levels then the pragmatic choice is to change the intervention. This iterative process continues until the student is making adequate progress. Across all phases of this process, data about student performance and treatment integrity should be the primary consideration in making decisions about continuing or changing an intervention. Basing decisions on only one source of data increases the risk of error.

Citations

Detrich, R. (2013). Innovation, implementation science, and data-based decision making: Components of successful reform. In M. Murphy, S. Redding, & J. Twyman (Eds.), Handbook on Innovations in Learning (pp. 33–49). Charlotte, NC: Information Age Publishing.

Sanetti, L. M., & Fallon, L. M. (2011). Treatment integrity assessment: How estimates of adherence, quality, and exposure influence interpretation of implementation. Journal of Educational and Psychological Consultation, 21(3), 209–232.

 

View this Overview as PDF 

 

Publications

TITLE
SYNOPSIS
CITATION
Treatment Integrity: Fundamental to Education Reform

To produce better outcomes for students two things are necessary: (1) effective, scientifically supported interventions (2) those interventions implemented with high integrity.  Typically, much greater attention has been given to identifying effective practices.  This review focuses on features of high quality implementation.

Detrich, R. (2014). Treatment integrity: Fundamental to education reform. Journal of Cognitive Education and Psychology, 13(2), 258-271.

Innovation, Implementation Science, and Data-Based Decision Making: Components of Successful Reform

Schools are often expected to implement innovative instructional programs.  Most often these initiatives fail because what we know from implementation science is not considered as part of implementing the initiative.  This chapter reviews the contributions implementation science can make for improving outcomes for students.

Detrich, R. Innovation, Implementation Science, and Data-Based Decision Making: Components of Successful Reform. Handbook on Innovations in Learning, 31.

Treatment Integrity: A Fundamental Unit of Sustainable Educational Programs.

Reform efforts tend to come and go very quickly in education. This paper makes the argument that the sustainability of programs is closely related to how well those programs are implemented.

Detrich, R., Keyworth, R. & States, J. (2010). Treatment Integrity: A Fundamental Unit of Sustainable Educational Programs. Journal of Evidence-Based Practices for Schools, 11(1), 4-29.

 

Data Mining

TITLE
SYNOPSIS
CITATION
How does performance feedback affect the way teachers carry out interventions?
This analysis examined the impact of performance feedback on the quality of implementation of interventions.
Detrich, R. (2015). How does performance feedback affect the way teachers carry out interventions? Retrieved from how-does-performance-feedback.
How long do reform initiatives last in public schools?
This analysis examined sustainability of education reforms in education.
Detrich, R. (2015). How long do reform initiatives last in public schools? Retrieved from how-long-do-reform.
How often are treatment integrity measures reported in published research?
This analysis examined the frequency that treatment integrity is reported in studies of research-based interventions.
Detrich, R. (2015). How often are treatment integrity measures reported in published research? Retrieved from how-often-are-treatment.
How well are Interventions Implemented in Educational Settings?
This analysis examined two studies to understand the reliability of self-reporting of practitioners implementing interventions.
Detrich, R. (2015). How well are Interventions Implemented in Educational Settings? Retrieved from how-well-are-interventions.

 

Presentations

TITLE
SYNOPSIS
CITATION
Treatment Integrity and Program Fidelity: Necessary but Not Sufficient to Sustain Programs
If programs are to sustain they must be implemented with integrity. If there is drift over time, it raises questions about whether the program is sustaining or has been substantially changed.
Detrich, R. (2008). Treatment Integrity and Program Fidelity: Necessary but Not Sufficient to Sustain Programs [Powerpoint Slides]. Retrieved from 2008-aba-presentation-ronnie-detrich.
Treatment Integrity: A Fundamental Component of PBS
School-wide initatives have to be well implemented if there is to be any benefit. This talk describes methods for assuring high levels of treatment integrity.
Detrich, R. (2008). Treatment Integrity: A Fundamental Component of PBS [Powerpoint Slides]. Retrieved from 2008-apbs-txint-presentation-ronnie-detrich.
Treatment Integrity: Necessary by Not Sufficient for Improving Outcomes
Treatment integrity is necessary to improve outcomes but it is not sufficient. It is also necessary to implement scientifically supported interventions.
Detrich, R. (2015). Treatment Integrity: Necessary by Not Sufficient for Improving Outcomes [Powerpoint Slides]. Retrieved from 2015-ebpindisabilities-txint-presentation-ronnie-detrich.

 

Student Research

TITLE
SYNOPSIS
CITATION
Effects of a problem solving team intervention on the problem-solving process: Improving concept knowledge, implementation integrity, and student outcomes.
This study evaluated the effects of a problem solving intervention package that included problem-solving information, performance feedback, and coaching in a student intervention planning protocol.
Vaccarello, C. A. (2011). Effects of a problem solving team intervention on the problem-solving process: Improving concept knowledge, implementation integrity, and student outcomes. Retrieved from student-research-2011.
TITLE
SYNOPSIS
CITATION
Enhancing Adherence to a Problem Solving Model for Middle-School Pre-Referral Teams: A Performance Feedback and Checklist Approach

This study looks at the use of performance feedback and checklists to improve middle-school teams problem solving.

Bartels, S. M., & Mortenson, B. P. (2006). Enhancing adherence to a problem-solving model for middle-school pre-referral teams: A performance feedback and checklist approach. Journal of Applied School Psychology, 22(1), 109-123.

Improving Mathematical Problem Solving in Grades 4 Through 8.

This practice guide provides five recommendations for improving students’ mathematical problem solving in grades 4 through 8. The manual is geared toward teachers, math coaches, other educators, and curriculum developers who want to improve the mathematical problem solving of students.

Clearinghouse, W. W. Improving Mathematical Problem Solving in Grades 4 Through 8. U.S. Department of Education, Institute of Education Science (IES) NCEE 2012-4055.

Developing Curriculum-Based Measurement Systems for Data-Based Special Education Problem Solving

This paper provides procedures for developing curriculum-based measurement systems in special education problem solving.

Deno, S. L., & Fuchs, L. S. (1987). Developing Curriculum-Based Measurement Systems for Data-Based Special Education Problem Solving. Focus on Exceptional Children, 19(8), 1-16.

Sustainability: An Enduring Commitment to Success

The infrastructure supporting organizational transformation to a problem-solving system occurred on two levels, global and local. The reform effort is described in four phases.

Grimes, J., Kurns, S., Tilly, W. D., & II, I. (2006). Sustainability: An enduring commitment to success. School Psychology Review35(2), 224.

Creating A Problem-Solving Culture: Exploring Problem Resolution In The Workplace

This paper examines and solutions associated with effective problem solving.

Marone, M. and Blauth, C. (2004). Creating A Problem-Solving Culture: Exploring Problem Resolution In The Workplace. Achieveglogal, Inc.

Demanding Medical Excellence: Doctors and Accountability in The Information Age.

This book reveals how the information revolution is changing the way doctors make decisions. Michael Millenson, a three-time Pulitzer Prize nominee as a health-care reporter for the Chicago Tribune, illustrates serious flaws in contemporary medical practice and shows ways to improve care and save tens of thousands of lives.

Millenson, M. L. (2018). Demanding medical excellence: Doctors and accountability in the information age. University of Chicago Press.

The Problem-Solving Power Of Teachers

This article examines practical strategies to address problems teacher are confronted with in the classroom.

Sacks, A. (2013). The problem-solving power of teachers. Educational Leadership, 71(2), 18-22.

Interpersonal problem solving as a mediator of behavioral adjustment in preschool and kindergarten children.

This is an experimental test of the mediating function of interpersonal cognitive problem solving skills on behavioral adjustment in preschool and kindergarten children.

Shure, M. B., & Spivack, G. (1981). Interpersonal problem solving as a mediator of behavioral adjustment in preschool and kindergarten children. Journal of Applied Developmental Psychology, 1(1), 29-44.

Troubleshooting Behavioral Interventions: A Systematic Process for Finding and Eliminating Problems

This article describes a systematic process for finding and resolving problems with classroom-based behavioral interventions in schools. 

Witt, J. C., VanDerHeyden, A. M., & Gilbertson, D. (2004). Troubleshooting behavioral interventions: A systematic process for finding and eliminating problems. School Psychology Review33, 363-383.

TITLE
SYNOPSIS
Center for Teaching

Resources provided by the Vanderbilt University Center for Teaching provides resources to assist teachers become better problem solvers.

Center for Teaching Excellence

Resources provided by the University of Waterloo Centre for Teaching Excellence provides resources to assist teachers become better problem solvers.

Teaching Excellence & Educational Innovation: Solve a Teaching Problem

This site provides practical strategies to address teaching problems across the disciplines. These strategies are firmly grounded in educational research and learning principles.

Back to Top