Approaches to Increasing Treatment Integrity
View this Overview as PDF
There are two general approaches to increasing treatment integrity. Educators can arrange either (a) antecedent conditions so that interventions are more likely to be implemented well or (b) consequences to encourage high-quality implementation. In reality, both approaches are likely necessary. Staff training is the most common antecedent approach, and performance feedback the most common consequent strategy. These and other methods are discussed below.
Staff training. Although staff training is easily the most common approach to ensuring that those responsible for implementation know what is expected of them, it is a very broad category and not all approaches are equally effective. Joyce and Showers (2002) completed a systematic review of professional development in education; the data are summarized in Figure 1below.
Figure 1. The effectiveness of various approaches to staff training
If you accept that the primary function of any effort to improve the quality of implementation is ultimately measured by the degree that the skills taught in training actually occur in the classroom, then, as Figure 1 illustrates, the most effective means of ensuring implementation is an approach that involves coaching and feedback in the classroom. The usual approaches to staff training, which include lecture, demonstration, and behavioral rehearsal without coaching in class, result in very little generalization to the classroom.
Training in the details of an intervention is necessary, but it is not sufficient to ensure high levels of treatment integrity. Much of the treatment integrity literature suggests that soon after training ends and implementation begins, treatment integrity begins to decline, even if implementers have demonstrated they can implement the intervention with 100% accuracy during training (Mortenson & Witt, 1998; Noell, Duhon, Gatti, & Connell, 2002; Noel et al., 2005).
Video modeling. In video modeling, a teaching approach in which a competent person is recorded performing a skill, the learner watches the video and then imitates the video model. Video modeling has been used to increase the quality of implementation of intervention plans in a small number of studies (Collins, Higbee, & Salzberg, 2009; DiGennaro-Reed, Codding, Catania, & Maguire, 2010; Hawkins & Heflin, 2011). All of these studies focused on the adherence dimension of treatment integrity. Hawkins and Heflin used video modeling and visual performance feedback to increase teachers’ use of behavior-specific praise. The results of this study are a bit unclear. Because video modeling and visual performance feedback were combined in a package to increase treatment integrity, sorting out the effects of the two elements is difficult. Although the combination is problematic from a research perspective, from a practice perspective the important effect is that the package increased treatment integrity. DiGennaro-Reed and colleagues (2010) examined the effects of video modeling on implementation of behavioral interventions and found that it produced improved but variable effects. When performance feedback was added to video modeling, treatment integrity was high and stable. These data suggest that performance feedback may be necessary to maximize the benefits of video modeling. In their investigation of video modeling as a means to improve the implementation of a problem-solving intervention, Collins and colleagues (2009) reported that video modeling alone resulted in stable performance across time and generalized to novel situations. The reason for the difference in outcomes needs to be examined. However, from a practice perspective video modeling has the potential to be an effective and efficient approach to increasing treatment integrity.
Among the efficiencies of video modeling is that once a standard video model of implementation is developed, it can be applied across individual teachers to increase implementation. Performance feedback can be added as needed. It may be that not all teachers will require feedback to maintain stable performance. Another advantage of video modeling is that individual teachers can view the video as needed without having to coordinate their schedules with other teachers and staff. The addition of performance feedback increases the cost, but it may be less costly than a performance feedback system without video modeling. If not all teachers require feedback, then the costs will be significantly reduced compared with a performance feedback only system.
Increasing motivation. Often, implementers are not motivated to implement an intervention because they find it unacceptable in some way. Extensive literature on treatment acceptability suggests that variables such as effort and time required to implement as well as compatibility of the intervention with the implementer’s perspective about appropriate intervention can influence the acceptability of interventions (Elliott, 1988; Miltenberger, 1990; Reimers, Wacker, & Koeppl, 1987). Any of these variables can influence the implementer’s willingness to implement the intervention as planned.
Contextual fit (a measure of how well an intervention integrates into the existing structures and routines of a classroom as well as the resources required to implement) has been described as a critical feature if interventions are to be implemented with integrity (Albin, Lucyshyn, Horner, & Flannery, 1996; Detrich, 1999). In a review of variables influencing the adoption of interventions, Riley-Tillman and Chafouleas (2003) argued that interventions requiring small adaptations are more likely to be adopted and sustained than interventions requiring significant changes to existing systems.
In a related study by Benazzi, Horner, and Good (2006), the technical adequacy and contextual fit of interventions developed by intervention teams with no behavior specialists on the team, by behavior specialists alone, and by teams that included a behavior specialist were assessed. The results were that technical adequacy was rated high if the behavior specialist alone or a team with a behavior specialist developed the intervention. Contextual fit was rated high when teams without a behavior specialist or teams with a behavior specialist developed the intervention. Plans developed by teams without a behavior specialist or teams with a behavior specialist were preferable to intervention plans developed by a behavior specialist alone. Although this study did not directly measure treatment integrity, it suggested that plans developed by those who have an understanding of the culture of the school and the classroom are more acceptable and therefore more likely to be implemented than plans that do not attend to these contextual variables. These data suggest that increasing contextual fit increases the motivation to implement, but this aspect needs to be directly and empirically evaluated.
Implementation planning. One antecedent approach to increasing contextual fit is implementation planning (Sanetti, Collier-Meek, Long, Kim, & Kratochwill, 2014). It involves identifying barriers and planning logistically for how an intervention will be implemented. The teacher responsible for implementation is very involved in the planning process, and that teacher and the intervention specialist such as a behavior analyst make decisions about how to adapt interventions to better fit the local context. In a study evaluating the implementation planning process, Sanetti and colleagues (2014) reported that both adherence and quality increased following implementation planning. In this instance, implementation planning was introduced following training and low levels of implementation. In practice, it is appropriate to use the implementation planning process when an intervention is initially being developed, thus ensuring that logistical issues and barriers are addressed and that implementation is high from the beginning of the intervention.
Choice. Another technique for increasing motivation to implement with integrity is to let the implementer choose the components of the intervention. Anderson and Daly (2013) allowed teachers to choose intervention components from a list of evidence-based strategies and to develop and implement their own interventions as well as an intervention developed by an expert. Then they compared the level of treatment integrity for the teacher-developed intervention with that for the intervention developed by an expert. Treatment integrity and student outcomes were better when the teachers implemented the intervention they developed compared with the expert-developed intervention.
A body of literature suggests that choice has reinforcing effects (Vaughn & Horner, 1997). It may well be that giving implementers a menu of options will be sufficiently reinforcing to increase the level of treatment integrity. At the very least, implementers are likely to choose elements of interventions that are the best contextual fit for their setting. In a study by Kern and colleagues (Kern, Childs, Dunlap, Clarke, & Falk, 1994), teachers were allowed to select intervention strategies that had been shown through preference assessments to influence student behavior. The teachers selected interventions that best fit their classroom routines and structures, and these teacher-selected interventions resulted in improved student behavior. There were no direct measures of treatment integrity in this study, so how well the teachers implemented the intervention is unknown, although it can be argued they implemented it well enough to have a positive impact on student behavior.
These data, along with the Anderson and Daly (2013) study, lead to the conclusion that those responsible for implementing an intervention should be involved in developing it. This likely has two benefits. It increases the face validity of the intervention, and it probably makes the intervention a better contextual fit. Elliott (1988) suggested that interventions are rated more acceptable if they are perceived to be effective. Among the variables that influence the perception of effectiveness is the degree to which an intervention is consistent with the implementer’s perspectives about appropriate treatment.
Performance feedback. The most common strategy for increasing treatment integrity is performance feedback (Auld, Belfiore, & Scheeler, 2010; Barton, Pribble, & Chen, 2013; Burns, Peters, & Noell, 2008; Casey & McWilliam, 2011; Noell & Gansle, 2014). Feedback has been used to increase single elements of an intervention plan such as differential reinforcement (Auld, Belifore, & Scheeler, 2010); to improve the overall implementation of academic interventions (Mortenson & Witt, 1998); to improve the implementation of behavior support plans (Codding, Feinberg, Dunn, & Pace 2005); and to improve the implementation of decision protocols by school-based problem-solving teams (Bartels & Mortenson, 2005; Burns, Peters, & Noell, 2008; Duhon, Mesmer, Gregerson, & Witt, 2009).
There are several effective means for delivering feedback: face-to-face (Mortenson & Witt, 1998); email (Barton, Pribble, & Chen, 2013); and graphed (Zoder-Martell et al., 2013). Some data suggest that verbal feedback paired with graphed feedback results in better implementation than verbal feedback alone (Sanetti, Luiselli, & Handler, 2007; Zoder-Martell et al., 2013).
Although performance feedback is unquestionably an effective means for increasing the quality of implementation of interventions, in service settings it poses some potential problems. As a basis for feedback, it usually requires direct observation of those responsible for implementation. Implementation in a public school requires consideration of which personnel should observe and give feedback. One of the challenges is the frequency with which feedback must be given. A number of studies suggest that feedback once per week is sufficient to maintain high levels of treatment integrity (Mortenson & Witt, 1998; Noell et al., 2005). In most of the published studies on performance feedback, external consultants provided the feedback. One study examined the effects of performance feedback provided by school personnel (Sanetti, Fallon, & Collier-Meek, 2013). If school personnel are to provide feedback to teachers, protocols demand consistent feedback. These data point to the need for a system that makes certain those responsible for monitoring treatment integrity are following protocols. Failure to do results in poor implementation of the performance monitoring system, which in turn results in inadequate implementation of the intervention by teachers. (A systems approach will be discussed in more detail in A Multitiered System of Support for Teachers, below.)
Coaching in the Classroom. In many ways, coaching in the classroom is a continuation of training with coaching, which occurs in an instructional setting removed from the implementation setting. Coaching in the classroom also involves performance feedback. Knight (2007) provided a clear description of what is required to make coaching effective. Joyce and Showers (2002) showed that training without coaching results in very little impact in the classroom, suggesting that coaching in the classroom is an essential component of training. Reinke, Stormont, Herman, and Newcomer (2014) demonstrated the necessity of performance feedback as part of the coaching process. In this study, teachers who received more performance feedback had higher levels of implementation than teachers who received less feedback. Similarly, teachers who received more coaching in the classroom after a low baseline level of implementation achieved higher levels of implementation than teachers who received less coaching.
Just as with performance feedback, coaching in the classroom has limitations. It is resource intensive, and most schools do not have a sufficient pool of personnel able to coach. The personnel constraints may limit a school district’s ability to scale up coaching in the classroom. Knight (2007) described coaching as a partnership between the coach and the teacher. An effective coach requires two broad skill sets. First, the coach must have the technical expertise to problem solve all of the issues that arise during implementation in a classroom. Second, the coach must have the necessary social influence skills to form a partnership with and to be perceived as credible by the teacher (Knight, 2007; Rogers, 2010). The benefits of coaching are minimized when the coach is not perceived as technically competent and credible.
Self-monitoring. Teachers self-monitoring their implementation has considerable appeal because, if effective, it is a relatively inexpensive approach to increasing treatment integrity. By comparison, both performance feedback and coaching are relatively high cost but have a well-established record for effectiveness, at least under research conditions. Self-monitoring is among the less well-researched strategies that hold promise. To date, the research literature has not provided empirical support for this strategy, although some data suggest ways of improving the effectiveness of self-monitoring.
Sanetti and colleagues (2013) evaluated the efficacy of the daily report card as a means of increasing treatment integrity. They compared effects of teachers verbally reporting their implementation versus providing a written self-report. These reports of implementation occurred either daily or weekly. Teachers who reported daily had higher levels of treatment integrity than those who reported weekly, but the differences were not statistically significant. Similarly, written reports of implementation resulted in slightly higher levels of treatment integrity than verbal reports, but, again, the results were not statistically significant. In spite of these results, self-evaluation approaches such as the daily report card continue to hold promise. The data from this study suggest that frequent, written evaluations may ultimately prove useful, but more research needs to be conducted before educators add self-evaluation strategies to their empirically supported efforts to increase treatment integrity.
A Multitiered System of Support for Teachers
Among the challenges of ensuring that interventions are implemented with integrity are these two factors: (a) Often, many individuals are responsible for implementation, and (b) limited resources are available to assess how well interventions are being implemented. One promising systems approach to ensuring higher levels of treatment integrity is a multitiered system of support for teachers (Myers, Simonsen, & Sugai, 2011; Sanetti & Collier-Meek, 2015). In this approach, all teachers are exposed to some effort to ensure high-quality implementation (tier 1), most often staff training. No additional action is required for teachers who respond to this level of support and implement with integrity. For those teachers who do not implement with adequate integrity, a tier 2 intervention, such as performance feedback or implementation planning, is put in place. Finally, some teachers will require an even more intensive intervention (tier 3), such as coaching, in order to implement with sufficient integrity. The appeal of this approach is that teachers receive only the level of support they need to be effective. The presumption is that as the interventions to support teachers increase in intensity across tiers of support, fewer teachers will require services.
In the study by Sanetti and Collier-Meek (2015), two of the six teachers required only tier 1 support. An additional two teachers were able to implement with integrity following tier 2 support. Finally, two teachers required the most intensive level of intervention (tier 3) to implement with integrity. Such an approach allows resources to be allocated based on identified need. This type of data-based, systematic approach to ensuring treatment integrity allows schools and school districts to scale up monitoring and to influence treatment integrity by using resources in an efficient and effective manner.
An implicit assumption of the multitiered system of support is that teachers must implement the intervention with integrity and those supporting teachers through staff training and other efforts must implement the support plan with integrity. There is some evidence that support plans are not always implemented with integrity (Sanetti, Fallon, & Collier-Meek, 2013). If this is the case, then it will be necessary for someone in the system to monitor the integrity of support plan implementation. Of course, then some part of the system will have to be responsible for making sure that the support plan for those supporting the teachers is implemented with integrity. Ultimately, treatment integrity is a systems problem and can only be solved by ensuring that each level of the system is involved in some capacity to make certain that interventions are implemented well in the classroom.
Detrich (2013) has proposed a data-based systems approach that involves all levels of the system to maintain high quality of implementation. As shown in Figure 5, two types of data are required in a systems approach to treatment integrity.
Figure 5. A data-based systems approach to treatment integrity
Data about student performance rolls up into larger and larger units of aggregated data. The classroom teacher is concerned about the performance of individual students in his or her class and the class as a whole. At the school level, the concern is with the performance of each individual class and the school. These data can be meaningfully interpreted only if something is known about how well the classroom teacher implemented the interventions in the classroom and how well those responsible for supporting implementation by the teacher implemented the support plan. This same logic applies across all levels of the system.
Whereas data about student performance rolls up across levels of the system, data about quality of implementation flows down across the different levels of the system. The state education agency provides data to the school district. At the district level, data is used as feedback to schools about how well they are implementing the teacher support plan. The school provides data to classroom teachers about how well they are implementing interventions in their classes.
Both student performance data and treatment integrity data are necessary. Student performance data without treatment integrity data do not allow us to make any conclusions about student performance because we do not know how well the intervention was implemented. Treatment integrity data without student performance data do not allow us to make any decisions about the effectiveness of the intervention.
This systems approach highlights the fact that what happens in an individual classroom is the responsibility of every level of the system. If any level of the system fails to support high-quality implementation, then it is likely that student performance will be negatively impacted and the system will fail in its primary responsibility to students.
Albin, R. W., Lucyshyn, J. M., Horner, R. H., & Flannery, K. B. (1996). Contextual fit for behavioral support plans: A model for “goodness of fit.” In L. K. Koegel, R. L. Koegel, & G. Dunlap (Eds.) Positive behavioral support: Including people with difficult behavior in the community (pp. 81–98). Baltimore, MD: Paul H. Brookes.
Andersen, M., & Daly, E. J. (2013). An experimental examination of the impact of choice of treatment components on treatment integrity. Journal of Educational and Psychological Consultation, 23(4), 231–263.
Auld, R. G., Belfiore, P. J., & Scheeler, M. C. (2010). Increasing pre-service teachers’ use of differential reinforcement: Effects of performance feedback on consequences for student behavior. Journal of Behavioral Education, 19(2), 169–183.
Bartels, S. M., & Mortenson, B. P. (2005). Enhancing adherence to a problem-solving model for middle-school pre-referral teams: A performance feedback and checklist approach. Journal of Applied School Psychology, 22(1), 109–123.
Barton, E. E., Pribble, L., & Chen, C.-I. (2013). The use of e-mail to deliver performance-based feedback to early childhood practitioners. Journal of Early Intervention, 35(3), 270–297.
Benazzi, L., Horner, R. H., & Good, R. H. (2006). Effects of behavior support team composition on the technical adequacy and contextual fit of behavior support plans. Journal of Special Education, 40(3), 160–170.
Burns, M. K., Peters, R., & Noell, G. H. (2008). Using performance feedback to enhance implementation fidelity of the problem-solving team process. Journal of School Psychology, 46(5), 537–550.
Casey, A. M., & McWilliam, R. A. (2011). The characteristics and effectiveness of feedback interventions applied in early childhood settings. Topics in Early Childhood Special Education, 31(2), 68–77.
Codding, R. S., Feinberg, A. B., Dunn, E. K., & Pace, G. M. (2005). Effects of immediate performance feedback on implementation of behavior support plans. Journal of Applied Behavior Analysis, 38(2), 205–219.
Collins, S., Higbee, T. S., & Salzberg, C. L. (2009). The effects of video modeling on staff implementation of a problem-solving intervention with adults with developmental disabilities. Journal of Applied Behavior Analysis, 42(4), 849–854.
Detrich, R. (1999). Increasing treatment fidelity by matching interventions to contextual variables within the educational setting. School Psychology Review, 28(4), 608–620.
Detrich, R. (2013). Innovation, implementation science, and data-based decision making: Components of successful reform. In M. Murphy, S. Redding, & J. Twyman (Eds.). Handbook on Innovations in Learning (pp. 33–49). Charlotte, NC: Information Age Publishing.
DiGennaro-Reed, F. D., Codding, R., Catania, C. N., & Maguire, H. (2010). Effects of video modeling on treatment integrity of behavioral interventions. Journal of Applied Behavior Analysis, 43(2), 291–295.
Duhon, G. J., Mesmer, E. M., Gregerson, L., & Witt, J. C. (2009). Effects of public feedback during RTI team meetings on teacher implementation integrity and student academic performance. Journal of School Psychology, 47(1), 19–37.
Elliott, S. N. (1988). Acceptability of behavioral treatments: A review of variables that influence treatment selection. Professional Psychology: Research and Practice, 19(1), 68–80.
Hawkins, S. M., & Heflin, L. J. (2011). Increasing secondary teachers’ behavior-specific praise using a video self-modeling and visual performance feedback intervention. Journal of Positive Behavior Interventions, 13(2), 97–108.
Joyce, B. R., & Showers, B. (2002). Student achievement through staff development. Alexandria, VA: Association for Supervision and Curriculum Development (ASCD) Books.
Kern, L., Childs, K. E., Dunlap, G., Clarke, S., & Falk, G. D. (1994). Using assessment-based curricular intervention to improve the classroom behavior of a student. Journal of Applied Behavior Analysis, 27(1), 7-19.
Knight, J. (2007). Instructional coaching: A partnership approach to improving instruction. Thousand Oaks, CA: Corwin Press.
Miltenberger, R. G. (1990). Assessment of treatment acceptability: A review of the literature. Topics in Early Childhood Special Education, 10(3), 24–38.
Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a prereferral academic intervention. School Psychology Review, 27(4), 613–627.
Myers, D. M., Simonsen, B., & Sugai, G. (2011). Increasing teachers’ use of praise with a response-to-intervention approach. Education and Treatment of Children, 34(1), 35–59.
Noell, G. H., Duhon, G. J., Gatti, S. L., & Connell, J. E. (2002). Consultation, follow-up, and implementation of behavior management interventions in general education. School Psychology Review, 31(2), 217–234.
Noell, G. H., & Gansle, K. A. (2014). The use of performance feedback to improve intervention implementation in schools. In L. M. Sanetti & T. R. Kratochwill (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 161–184). Washington, DC: American Psychological Association.
Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., . . . Duhon, G. J. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34(1), 87–106.
Reimers, T. M., Wacker, D. P., & Koeppl, G. (1987). Acceptability of behavioral interventions: A review of the literature. School Psychology Review, 16(2), 212–227.
Reinke, W., Stormont, M., Herman, K., & Newcomer, L. (2014). Using coaching to support teacher implementation of classroom-based interventions. Journal of Behavioral Education, 23(1), 150–167.
Riley-Tillman, T. C., & Chafouleas, S. M. (2003). Using interventions that exist in the natural environment to increase treatment integrity and social influence in consultation. Journal of Educational & Psychological Consultation, 14(2), 139–156.
Rogers, E. M. (2010). Diffusion of innovations (5th ed.). New York, NY: Simon & Schuster.
Sanetti, L. M. H., Chafouleas, S. M., O’Keeffe, B. V., & Kilgus, S. P. (2013). Treatment integrity assessment of a daily report card intervention: A preliminary evaluation of two methods and frequencies. Canadian Journal of School Psychology, 28(3), 261–276.
Sanetti, L. M. H., & Collier-Meek, M. A. (2015). Data-driven delivery of implementation supports in a multi-tiered framework: A pilot study. Psychology in the Schools, 52(8), 815–828.
Sanetti, L. M. H., Collier-Meek, M. A., Long, A. C. J., Kim, J., & Kratochwill, T. R. (2014). Using implementation planning to increase teachers’ adherence and quality to behavior support plans. Psychology in the Schools, 51(8), 879–895.
Sanetti, L. M. H., Fallon, L. M., & Collier-Meek, M. A. (2013). Increasing teacher treatment integrity through performance feedback provided by school personnel. Psychology in the Schools, 50(2), 134–150.
Sanetti, L. M. H., Luiselli, J. K., & Handler, M. W. (2007). Effects of verbal and graphic performance feedback on behavior support plan implementation in a public elementary school. Behavior Modification, 31(4), 454–465.
Vaughn, B. J., & Horner, R. H. (1997). Identifying instructional tasks that occasion problem behaviors and assessing the effects of student versus teacher choice among these tasks. Journal of Applied Behavior Analysis, 30(2), 299–312.
Zoder-Martell, K., Dufrene, B., Sterling, H., Tingstrom, D., Blaze, J., Duncan, N., & Harpole, L. (2013). Effects of verbal and graphed feedback on treatment integrity. Journal of Applied School Psychology, 29(4), 328–349.
View this Overview as PDF