Education Drivers

Program Fidelity

Publications

TITLE
SYNOPSIS
CITATION
Seeking the Magic Metric: Using Evidence to Identify and Track School System Progress

This paper discusses the search for a “magic metric” in education: an index/number that would be generally accepted as the most efficient descriptor of school’s performance in a district.

Celio, M. B. (2013). Seeking the Magic Metric: Using Evidence to Identify and Track School System Quality. In Performance Feedback: Using Data to Improve Educator Performance (Vol. 3, pp. 97-118). Oakland, CA: The Wing Institute.

Treatment Integrity in the Problem Solving Process

The usual approach to determining if an intervention is effective for a student is to review student outcome data; however, this is only part of the task. Student data can only be understood if we know something about how well the intervention was implemented. Student data without treatment integrity data are largely meaningless because without knowing how well an intervention has been implemented, no judgments can be made about the effectiveness of the intervention. Poor outcomes can be a function of an ineffective intervention or poor implementation of the intervention. Without treatment integrity data, the is a risk that an intervention will be judged as ineffective when, in fact, the quality of implementation was so inadequate that it would be unreasonable to expect positive outcomes.

Detrich, R., States, J. & Keyworth, R. (2017). Treatment Integrity in the Problem Solving Process. Oakland, Ca. The Wing Institute.

 

Sustainability of evidence-based programs in education

This paper discusses common elements of successfully sustaining effective practices across a variety of disciplines.

Fixsen, D. L., Blase, K. A., Duda, M., Naoom, S. F., & Van Dyke, M. (2010). Sustainability of evidence-based programs in education. Journal of Evidence-Based Practices for Schools, 11(1), 30-46.

 

Data Mining

TITLE
SYNOPSIS
CITATION
Is there empirical research to validate the use of prereferral intervention teams (PIT) to reduce special education referrals, achieve gains, or improve student conduct?
This is a review of a meta-analysis of Prereferral Intervention Teams on student and system outcomes.
States, J. (2011). Is there empirical research to validate the use of prereferral intervention teams (PIT) to reduce special education referrals, achieve gains, or improve student conduct? Retrieved from is-there-empirical-research.

 

Presentations

TITLE
SYNOPSIS
CITATION
Seeking the Magic Metric: Using Evidence to Identify and Track School System Progress

This paper discusses the search for a “magic metric” in education: an index/number that would be generally accepted as the most efficient descriptor of school’s performance in a district.

Celio, MB. (2011). Seeking the Magic Metric: Using Evidence to Identify and Track School System Progress [Powerpoint Slides]. Retrieved from 2011-wing-presentation-mary-beth-celio.

A Systems Approach to Feedback: What You Need to Know and Who Needs
This paper looks at feedback as a powerful systems approach to improving the performance of both student and school faculty.
States, J. (2011). A Systems Approach to Feedback: What You Need to Know and Who Needs [Powerpoint Slides]. Retrieved from 2011-calaba-presentation-jack-states.
Feedback as Education Reform: What We Know
This paper examines the power of feedback as a strategy for improving student performance. Types of feeback are explored building from student and teacher performance that can be aggregated to create a systems wide feedback tool.
States, J. (2011). Feedback as Education Reform: What We Know [Powerpoint Slides]. Retrieved from 2011-aba-presentation-jack-states.
TITLE
SYNOPSIS
CITATION
Introduction: Proceedings from the Wing Institute’s Sixth Annual Summit on Evidence-Based Education: Performance Feedback: Using Data to Improve Educator Performance.

This book is compiled from the proceedings of the sixth summit entitled “Performance Feedback: Using Data to Improve Educator Performance.” The 2011 summit topic was selected to help answer the following question: What basic practice has the potential for the greatest impact on changing the behavior of students, teachers, and school administrative personnel?

States, J., Keyworth, R. & Detrich, R. (2013). Introduction: Proceedings from the Wing Institute’s Sixth Annual Summit on Evidence-Based Education: Performance Feedback: Using Data to Improve Educator Performance. In Education at the Crossroads: The State of Teacher Preparation (Vol. 3, pp. ix-xii). Oakland, CA: The Wing Institute.

 

 

Differences between graduates of 4-year and 5-year teacher preparation programs

A 10-year comparison of graduates from 4- and 5-year teacher education programs at the same institution revealed significant differences between graduates of the two programs. Limitations of the study and alternative explanations for these differences are discussed.

Andrew, M. D. (1990). Differences between graduates of 4-year and 5-year teacher preparation programs. Journal of Teacher Education, 41, 45–51

Can evidence-based prevention programs be sustained in community practice settings? The Early Risers' advanced-stage effectiveness trial

This study evaluated institutional sustainability of the Early Risers “Skills for Success” conduct problems prevention program.

August, G. J., Bloomquist, M. L., Lee, S. S., Realmuto, G. M., & Hektner, J. M. (2006). Can evidence-based prevention programs be sustained in community practice settings? The Early Risers’ advanced-stage effectiveness trial. Prevention Science7(2), 151-165.

Increasing Pre-service Teachers’ Use of Differential Reinforcement: Effects of Performance Feedback on Consequences for Student Behavior

Differential reinforcement of appropriate behavior is an important skill for classroom teachers. This study examined the use of performance feedback to increase the rate of differential reinforcement by pre-service teachers.

Auld, R. G., Belfiore, P. J., & Scheeler, M. C. (2010). Increasing Pre-service Teachers’ Use of Differential Reinforcement: Effects of Performance Feedback on Consequences for Student Behavior. Journal of Behavioral Education, 19(2), 169-183.

Finding the Balance: Program Fidelity and Adaptation in Substance Abuse Prevention: A State-of-the-Art Review

In this paper, the author surveys 117 published and unpublished studies spanning more than 25 years. He structures the literature review using a conceptual model shown as Program Implementation Stages

Backer, T. E. (2001). Finding the balance: Program fidelity and adaptation in substance abuse prevention: A state-of-the-art review. Rockville, MD: Center for Substance Abuse Prevention.

The fidelity‐adaptation debate: Implications for the implementation of public sector social programs

The modified Research, Development, and Diffusion (RD&D) model as exemplified by
change agents in federal organizations were examined as a viable strategy for
disseminating social program innovations.

Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., & Emshoff, J. G. (1987). The fidelity‐adaptation debate: Implications for the implementation of public sector social programs. American journal of community psychology15(3), 253-268.

Measurement of Fidelity in Psychiatric Rehabilitation

This paper review the promise of fidelity measures for advancing the research and practice of one area of mental health services, namely, psychiatric rehabilitation. This paper also provide a historical context for the development of fidelity measurement.

Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. W. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental health services research2(2), 75-87.

Using performance feedback to enhance implementation fidelity of the problem-solving team process

This study examines the importance of implementation integrity for problem-solving teams (PST) and response-to-intervention models.

Burns, M. K., Peters, R., & Noell, G. H. (2008). Using performance feedback to enhance implementation fidelity of the problem-solving team process. Journal of School Psychology, 46(5), 537-550.

Using performance feedback to enhance implementation fidelity of the problem-solving team process

The current study hypothesized that providing performance feedback, which has consistently been shown to increase implementation integrity, to PSTs would enhance the procedural integrity of the process.

Burns, M. K., Peters, R., & Noell, G. H. (2008). Using performance feedback to enhance implementation fidelity of the problem-solving team process. Journal of School Psychology46(5), 537-550.

Seeking the Magic Metric: Using Evidence to Identify and Track School System Quality

This paper discusses the search for a “magic metric” in education: an index/number that would be generally accepted as the most efficient descriptor of school’s performance in a district.

Celio, M. B. (2013). Seeking the Magic Metric: Using Evidence to Identify and Track School System Quality. In Performance Feedback: Using Data to Improve Educator Performance (Vol. 3, pp. 97-118). Oakland, CA: The Wing Institute.

Effects of immediate performance feedback on implementation of behavior support plans, 2005

The purpose of this study is to examine the effects of feedback on treatment integrity for implementing behavior support plans.

Codding, R. S., Feinberg, A. B., Dunn, E. K., & Pace, G. M. (2005). Effects of immediate performance feedback on implementation of behavior support plans. Journal of Applied Behavior Analysis, 38(2), 205-219.

Program integrity in primary and early secondary prevention: are implementation effects out of control?

The authors examined the extent to which program integrity (i.e., the degree to which programs were implemented as planned) was verified and promoted in evaluations of primary and early secondary prevention programs published between 1980 and 1994. 

Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: are implementation effects out of control?. Clinical psychology review18(1), 23-45.

Innovation, Implementation Science, and Data-Based Decision Making: Components of Successful Reform

Over the last fifty years, there have been many educational reform efforts, most of which have had a relatively short lifespan and failed to produce the promised results. One possible reason for this is for the most part these innovations have been poorly implemented. In this chapter, the author proposes a data-based decision making approach to assuring high quality implementation.

Detrich, R. Innovation, Implementation Science, and Data-Based Decision Making: Components of Successful Reform. In M. Murphy, S. Redding, and J. Twyman (Eds). Handbook on Innovations in Learning, 31. Charlotte, NC: Information Age Publishing

The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children

This article reviews implementation issues in prevention trials and specifically highlights the study of implementation in the 34 programs determined to be effective in a recent review conducted by the Prevention Research Center for the Center for Mental Health Services.

Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation11(2), 193-221.

Issues in Disseminating and Replicating Effective Prevention Programs

a brief overview of findings from the Blueprints for Violence Prevention-Replication Initiative is presented, identifying factors that enhance or impede a successful implementation of these programs.

Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science5(1), 47-53.

Sustainability of evidence-based programs in education

This paper discusses common elements of successfully sustaining effective practices across a variety of disciplines.

Fixsen, D. L., Blase, K. A., Duda, M., Naoom, S. F., & Van Dyke, M. (2010). Sustainability of evidence-based programs in education. Journal of Evidence-Based Practices for Schools, 11(1), 30-46.

Implementation Research: A Synthesis of the Literature

This is a comprehensive literature review of the topic of Implementation examining all stages beginning with adoption and ending with sustainability.

Fixsen, D. L., Naoom, S. F., Blase, K. A., & Friedman, R. M. (2005). Implementation research: A synthesis of the literature.

Evaluating Fidelity: Predictive Validity for a Measure of Competent Adherence to the Oregon Model of Parent Management Training

This article presents the Fidelity of Implementation Rating System (FIMP), an observation-based measure assessing competent adherence to the Oregon model of Parent Management Training (PMTO). 

Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior therapy36(1), 3-13.

A framework for building capacity for responsiveness to intervention

This commentary offers such a framework and then they consider how the articles constituting this special issue address the various questions posed in the framework. 

Fuchs, L. S., & Fuchs, D. (2006). A framework for building capacity for responsiveness to intervention. School Psychology Review35(4), 621.

Developing Leaders: The Importance--and the Challenges--of Evaluating Principal Preparation Programs

The authors have been evaluating the impact of five principal preparation programs in the United States on student outcomes. This information should be considered as one aspect of preparation program improvement and accountability. The study team lays out its recommendations in this policy paper. 

George W. Bush Institute & Education Reform Initiative. (2016). Developing Leaders: The Importance--and the Challenges--of Evaluating Principal Preparation Programs, Retrieved from https://gwbcenter.imgix.net/Resources/gwbi-importance-of-evaluating-principal-prep.pdf

Treatment integrity of school-based behavioral intervention studies: 1980-1990

This study reviewed the reporting rates of treatment integrity for school based interventions.

Gresham, F. M., & Gansle, K. A. (1993). Treatment integrity of school-based behavioral intervention studies: 1980-1990. School Psychology Review, 254. 

Treatment integrity in applied behavior analysis with children.

This study reviewed all intervention studies published between 1980-1990 in Journal of Applied Behavior Analysis in which children were the subjects of the study. The authors found that treatment integrity was reported in only 16% of the studies.

Gresham, F. M., Gansle, K. A., & Noell, G. H. (1993). Treatment ?integrity in ?applied behavior analysis with children. Journal of Applied ?Behavior Analysis, 26(2), 257-263.

Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented

The authors reviewed three learning disabilities journals between 1995-1999 to determine what percent of the intervention studies reported measures of treatment integrity. Only 18.5% reported treatment integrity measures.

Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., & Bocian, K. M. ?(2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented. Learning Disabilities ?Research & Practice, 15(4), 198-205.

Effects of Two Levels of Procedural Fidelity with Constant Time Delay on Children's Learning

The purpose of this study was to compare the effects of constant time delay delivered with high procedural fidelity to constant time delay with high procedural fidelity on all variables except delivery of the controlling prompt

Holcombe, A., Wolery, M., & Snyder, E. (1994). Effects of two levels of procedural fidelity with constant time delay on children's learning. Journal of Behavioral Education4(1), 49-73.

An Investigation of Concurrent Validity of Fidelity of Implementation Measures at Initial Years of Implementation.

The present study used cross-sectional data from 1,438 schools to examine relations between fidelity self-assessment and team-based fidelity measures in the first 4 years of implementation of School-Wide Positive Behavioral Interventions and Supports (SWPBIS). Results showed strong positive correlations between fidelity self-assessments and a team-based measure of fidelity at each year of implementation.

Khoury, C. R., McIntosh, K., & Hoselton, R. (2019). An Investigation of Concurrent Validity of Fidelity of Implementation Measures at Initial Years of Implementation. Remedial and Special Education40(1), 25-31.

 

High Versus Low Implementation of Instructional Support Teams: A Case for Maintaining Program Fidelity

In 1990, the Commonwealth of Pennsylvania implemented a statewide instructional support team (IST) process to provide prereferral assessment and intervention for at-risk students in 500 school districts. The current study examined the academic performance of students affected by this process as contrasted with other at-risk students who did not have access to it.

Kovaleski, J. F., Gickling, E. E., Morrow, H., & Swank, P. R. (1999). High versus low implementation of instructional support teams: A case for maintaining program fidelity. Remedial and Special Education20(3), 170-183.

Using Coaching to improve the Fidelity of Evidence-Based Practices: A Review of Studies

The authors conducted a comprehensive review of research to identify the impact of coaching on changes in preservice and in-service teachers’ implementation of evidence-based practices.

Kretlow, A. G., & Bartholomew, C. C. (2010). Using coaching to improve the fidelity of evidence-based practices: A review of studies. Teacher Education and Special Education33(4), 279-299.

Outcome for children with autism receiving early and intensive behavioral intervention in mainstream preschool and kindergarten settings

This study examined whether children receiving one year of EIBI (N = 35) would make larger gains in adaptive behaviors than a group of children receiving treatment as usual (TAU; N = 24).

McEachin, J. J., Smith, T., & Lovaas, O. I. (1993). Outcome in adolescence of autistic children receiving early intensive behavioral treatment. American Journal of Mental Retardation97, 359-372.

The Use of Weekly Performance Feedback to Increase Teacher Implementation of a Pre-referral Academic Intervention.

This study evaluated the effects of performance feedback on the implementation of a classroom intervention.

Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a prereferral academic intervention. School Psychology Review, 613-627. 

Literature Review of Principal Preparation Programs

This literature review highlights central elements of the residency component of school leader preparation programs by aggregating the results of studies conducted on existing principal preparation programs. 

New York City Leadership Academy. (2013). Literature Review of Principal Preparation Programs. Retrieved from https://www.nycleadershipacademy.org/wp-content/uploads/2018/06/residency-design-literature-review.pdf

Literature Review of Principal Preparation Programs

This literature review highlights central elements of the residency component of school leader preparation programs by aggregating the results of studies conducted on existing principal preparation programs. 

New York City Leadership Academy. (2013). Literature Review of Principal Preparation Programs. Retrieved from https://www.nycleadershipacademy.org/wp-content/uploads/2018/06/residency-design-literature-review.pdf

Increasing intervention implementation in general education following consultation: A comparison of two follow-up strategies.

This study compared the effects of discussing issues of implementation challenges and performance feedback on increasing the integrity of implementation. Performance feedback was more effective than discussion in increasing integrity.

Noell, G. H., & Witt, J. C. (2000). Increasing intervention implementation in general education following consultation: A comparison of two follow-up strategies. Journal of Applied Behavior Analysis, 33(3), 271.

Increasing Teacher Intervention Implementation in General Education Settings through Consultation and Performance Feedback

This study looks at treatment integrity with which general education teachers implemented a reinforcement based intervention designed to improve the academic performance of elementary school students.

Noell, G. H., Witt, J. C., Gilbertson, D. N., Ranier, D. D., & Freeland, J. T. (1997). Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12(1), 77.

Implementation of parent management training at the national level: The case of Norway

This article describes early aspects of the nationwide implementation of an evidence‐based program (EBP) in Norway and the design for studying program fidelity over time. 

Ogden, T., Forgatch, M. S., Askeland, E., Patterson, G. R., & Bullock, B. M. (2005). Implementation of parent management training at the national level: The case of Norway. Journal of Social Work Practice19(3), 317-329.

Treatment Integrity of Interventions With Children in the Journal of Positive Behavior Interventions: From 1999 to 2009

The authors reviewed all intervention studies published in the Journal of Positive Behavior Interventions between 1999-2009 to determine the percent of those studies that reported a measure of treatment integrity. Slightly more than 40% reported a measure of treatment integrity.

Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment Integrity of Interventions With Children in the Journal of Positive Behavior Interventions: From 1999 to 2009. Journal of Positive Behavior Interventions, 14(1), 29-46.

Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008

The authors reviewed four school psychology journals between 1995-2008 to estimate the percent of intervention studies that reported some measure of treatment integrity. About 50% reported a measure of treatment integrity.

Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40(1), 72-84.

A Consumer’s Guide to Evaluating a Core Reading Program Grades K-3: A Critical Elements Analysis

A critical review of reading programs requires objective and in-depth analysis. For these reasons, the authors offer the following recommendations and procedures for analyzing critical elements of programs.

Simmons, D. C., & Kame’enui, E. J. (2003). A consumer’s guide to evaluating a core reading program grades K-3: A critical elements analysis. Retrieved December19, 2006.

Scaling Up an Early Reading Program: Relationships Among Teacher Support, Fidelity of Implementation, and Student Performance Across Different Sites and Years

In this article, the authors address the following questions: How does the level of on-site technical assistance affect student outcomes? Do teachers’ fidelity of treatment implementation and their perceptions of school climate mediate effects on student performance?

Stein, M. L., Berends, M., Fuchs, D., McMaster, K., Sáenz, L., Yen, L., ... & Compton, D. L. (2008). Scaling up an early reading program: Relationships among teacher support, fidelity of implementation, and student performance across different sites and years. Educational Evaluation and Policy Analysis30(4), 368-388.

Fidelity of problem-solving implementation and relationship to student performance

This study examined the fidelity of problem-solving implementation by multidisciplinary
teams (MDTs) in 227 schools and the relationship to student outcomes.

Telzrow, C. F., McNamara, K., & Hollinger, C. L. (2000). Fidelity of problem-solving implementation and relationship to student performance. School Psychology Review29(3), 443.

Improving State Evaluation of Principal Preparation Programs

Intended for state officials involved in the assessment and approval of university and other programs to train future school principals, this report describes five design principles for effective program evaluation.

UCEA and New Leaders (2016). Improving state evaluation of principal preparation programs. Retrieved from: www.sepkit.org

From Policy to Practice: Evidence-based Practice and IDEIA
No Child Left Behind (NCLB) as federal policy required the use of scientifically based instruction to improve educational outcomes. It is a long way from policy to impacting what actually happens in a classroom. In this chapter, the author reviews what is known from implementation science to assure that policy becomes practice.
Detrich, R. (2008). From Policy to Practice: Evidence-based Practice and IDEIA. In Grigorenko, E. L. (Ed.) Educating Individuals with Disabilities: IDEIA 2004 and Beyond, 85. Springer Publishing Company
A Decade of Evidence-Based Education: Where Are We and Where Do We Need to Go?
No Child Left Behind (NCLB) emphasizing scientifically based instruction became law in 2001. In this paper, the authors review the progress that has been made in implementing NCLB and conclude that the greatest threat to its effectiveness is the quality of the implementation of the scientifically based instructional practices.
Detrich, R., & Lewis, T. (2012). A Decade of Evidence-Based Education: Where Are We and Where Do We Need to Go?. Journal of Positive Behavior Interventions, 1098300712460278.
Evolution of the Treatment Integrity Concept: Current Status and Future Directions
The article discusses treatment integrity.
Gresham, F. M. (2009). Evolution of the Treatment Integrity Concept: Current Status and Future Directions. School Psychology Review, 38(4), 533-540.
Toward Developing a Science of Treatment Integrity: Introduction to the Special Series
This article provides an overview of the current state of conceptual models of treatment integrity and suggest future directions for research, practice, and policy related to treatment integrity.
Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2009). Toward Developing a Science of Treatment Integrity: Introduction to the Special Series. School Psychology Review, 38(4), 445-459.
Balancing Fidelity With Flexibility and Fit: What Do We Really Know About Fidelity of Implementation in Schools
Treatment fidelity is critical for the effective implementation of evidence-based practices. It is assumed that evidence-based practices implemented with high fidelity will result in improved outcomes, whereas low fidelity will lead to poorer outcomes. This article discusses treatment fidelity as well as the implications for measuring and analyzing it in relation to student outcomes.
Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing Fidelity With Flexibility and Fit: What Do We Really Know About Fidelity of Implementation in Schools?. Exceptional Children, 79(2), 181-193.
Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005.
The authors reviewed all intervention studies published in the Journal of Applied Behavior Analysis between 1991-2005 that occurred in school settings to determine the percent of studies that reported treatment integrity measures. In this review, 30% of the studies reported treatment integrity.
McIntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the ?journal of applied behavior analysis 1991-2005. Journal of Applied Behavior Analysis, 40(4), 659-672.
Increasing teacher intervention implementation in general education settings through consultation and performance feedback
This study evaluated the effects of consultation and performance feedback in a public school setting.
Noell, G. H., Witt, J. C., Gilbertson, D. N., Ranier, D. D., & Freeland, J. T. (1997). ?Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12(1), 77-88.
Treatment Implementation Following Behavioral Consultation in Schools: A Comparison of Three Follow-up Strategies
This study examines teachers' implementation of treatment plans following consultation.
Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., ... & Duhon, G. J. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review.

No items found.

Back to Top