October 5, 2021
A review of the evidence for real-time performance feedback to improve instructional practice. Performance feedback, including real time performance feedback, has been implemented across many contexts including educational settings. Sinclair, Gesel, LeJeune, and Lemons (2020) evaluated 32 studies that met their inclusion criteria to determine the effectiveness of real-time performance feedback in improving the instructional practices of educators. Interestingly, all but two of the studies utilized bug-in-the-ear technology in which the teacher wore an ear piece and a coach provided immediate feedback to the teacher as implementation of an intervention was occurring. In this review, teachers implementing both academic instruction and behavior management interventions were considered. Based on this review, the authors concluded that real time performance feedback was an evidence-based practice and could be used as a method for improving the performance of pre-service teachers, teachers, and paraprofessionals. The bug-in-the-ear technology offers several advantages. First, it is less intrusive than other methods for providing real time feedback. With current technology, the coach can view implementation in the classroom without being in the classroom. Technologies such as Go Pro and Swivl have sufficient flexibility for the coach to get a good sample of what is happening in the classroom. A second advantage is that because the feedback is immediate it is more likely to be effective compared to when the feedback is delayed. Finally, the bug-in-the-ear technology is time saving because the feedback is delivered in real-time. Brief follow-up meetings to discuss issues related to the intervention can be scheduled. Interestingly, bug-in-the-ear technology has been around for decades but has been under-utilized in educational settings. An analysis of the barriers to utilizing this technology more broadly is warranted. The potential for impact on implementation of an intervention is significant.
Citation: Sinclair, A. C., Gesel, S. A., LeJeune, L. M., & Lemons, C. J. (2020). A review of the evidence for real-time performance feedback to improve instructional practice. The Journal of Special Education, 54(2), 90-100.
October 4, 2021
Electronically Delivered Support to Promote Intervention Implementation Fidelity: A Research Synthesis. Fundamental to any intervention outcome is the fidelity of implementation of the intervention. The ultimate goal of implementation science is to assure that innovations are implemented well enough for students to benefit. Failure to implement well can minimize the effectiveness of even the most powerful intervention. One of the challenges involved in insuring high quality implementation is that most approaches are resource intensive and often are not seen as feasible in school settings even though failure to achieve adequate implementation fidelity may result in a very poor benefit to cost ratio. One possible alternative is to utilize technology to reduce the resource demands. Fallon and colleagues (2021) conducted a systematic review to evaluate the effectiveness of technology-based supports to promote implementation fidelity. For the purposes of this review, “electronically delivered implementation supports (EDIS) was support delivered to an implementer electronically (e.g., via email, social media, video conferencing) for the purpose of improving educators’ implementation fidelity of a student intervention.” Fifteen studies met inclusion criteria and were judged to be of sufficient methodological rigor to warrant further analysis. The electronically delivered implementation supports ranged from video modeling, electronically delivered performance feedback, emailed intervention prompts, coaching via video conference, and online training modules. All of the studies were based on single participant designs. Since there are no agreed upon methods for calculating effect sizes for single participant designs, the authors calculated several different effect sizes (Tau-U, Standard Mean Difference, Hedges’ g, and a variation of Hedges’ g. In most of the studies, the effect sizes ranged from moderate to large regardless of the calculation method used. After completing the review, the authors provided guidance to educators about when to use the various methods of electronically delivered implementation supports. This article is a valuable resource to any educator considering implementing an intervention but is concerned about the resource requirements required for insuring high quality implementation. This article suggests that technology-based alternatives can be effective in supporting implementation and may reduce the overall demands on resources.
Citation: Fallon, L. M., Collier-Meek, M. A., Famolare, G. M., DeFouw, E. R., & Gould, K. M. (2020). Electronically Delivered Support to Promote Intervention Implementation Fidelity: A Research Synthesis. School Psychology Review, 1-16.
October 4, 2021
School-Based Interventions for Middle School Students With Disruptive Behaviors: A Systematic Review of Components and Methodology. Middle school students are more likely than elementary or high school students to be disruptive (Erikson & Gresham, 2019). This presents difficult problems for classroom teachers trying to provide instruction and maintain order in the classroom. It has been estimated that 2.5 hours per week are lost to disruptive behavior each week (Education Advisory Board, 2019). This level of disruptive also contributes to teacher turnover with 39% of teachers reporting that disruptive behavior was one of the primary reasons for resigning (Bettini et al., 2020). To address challenges presented by middle school students, Alperin and colleagues (2021) completed a systematic review to identify programs that had a positive effect on disruptive behavior and the characteristics of those programs. They identified 51 studies that met their inclusion criteria. Of those 51 studies, 40 of them specified the function of behavior (gain attention or escape demands) that the program addressed; 16 of the studies included a home-based component with 7 of the studies providing parent training; 22 of the interventions had a manual guiding the implementation of the intervention; and encouragingly, 42 of the studies assessed intervention implementation. Effect sizes for seven of the studies were computed for intervention that involved class-wide intervention strategies. The effects ranged from small to large across the studies. Fourteen of the studies evaluated skill acquisition for small groups or individuals and the effect sizes again ranged from small to large. Seven of the studies evaluated reinforcement strategies for reinforcement-based interventions for small groups or individuals and reported effect sizes that ranged from small to large. Two studies evaluated interventions for escape from demands for small groups or individuals. Both of these studies reported large effect sizes. The data from this study are important as they can provide guidance to educators seeking to reduce disruptive behavior of middle school students. Ultimately, the educators will have to consider the contextual fit for each of these interventions for the settings in which they work. This study narrows the range of options to those that have some demonstrated level of effectiveness rather than leaving the educator to choose from all available options.
Citation: Alperin, A., Reddy, L. A., Glover, T. A., Bronstein, B., Wiggs, N. B., & Dudek, C. M. (2021). School-Based Interventions for Middle School Students With Disruptive Behaviors: A Systematic Review of Components and Methodology. School Psychology Review, 1-26.
October 4, 2021
Scaling and Disseminating Brief Bullying Prevention Programming: Strengths, Challenges, and Considerations. One of the persistent problems in education and other human service disciplines is the research to practice gap (some would call it a chasm). In an effort to disseminate an effective bullying program (Free2B), Leff and colleagues applied the logic of Diffusion of Innovations (Rogers, 2003). This logic proposes that innovations are more likely to be adopted if the innovation has (1) a relative advantage over current practices, (2) is easy to use, (3) is compatible with the values, beliefs, experiences of the users, (4) can be implemented on a trial basis before large scale implementation, and (5) the opportunity for others to observe implementation and the effects of implementation. Leff and colleagues followed these recommendations in implementing the Free2B anti-bullying program in 40 middle schools. The authors concluded that it was an attractive alternative to many anti-bullying programs because the intervention was delivered in a school assembly that schools were already providing, so it required no additional time allocation. Additionally, the video format made the delivery very easy compared to school-wide programs that are more time and resource intensive. The students reported that it addressed important topics. Prior to implementation, Leff and colleagues presented pilot data to key stakeholders at the state’s Office of Safe Schools who were able to leverage adoption by schools across the state. In addition to measuring adoption they also measured the impact on students and founds positive effects across all measures.
Citations: Leff, S. S., Waasdorp, T. E., Paskewich, B. S., & Winston, F. K. (2020). Scaling and Disseminating Brief Bullying Prevention Programming: Strengths, Challenges, and Considerations. School Psychology Review, 1-15.
Rogers, E. M. (1962). Diffusion of innovations. New York: Free Press of Glencoe.
October 4, 2021
Overview of Professional Judgment. Educators make many decisions regarding services for students. Even when there is abundant evidence to guide their decisions, educators must use their judgment about what is appropriate in a given situation. Only on rare occasion does the available evidence perfectly match the service context of concern to the educator. To bridge the gap between research and local circumstance, the educator must make a series of judgments such as defining the problem, determining which evidence is relevant, and deciding which features of the local context are likely to require adaptations to the selected evidence-based intervention. Professional judgment is a cornerstone of evidence-based practice, as are best available evidence, stakeholder values, and the context in which services are provided. In this definition of evidence-based practice, the integration of these variables influences decisions. No one cornerstone can be substituted for the others. Judgment must be informed and constrained by the best available evidence, stakeholder values, and context.
Citation: Guinness, K., and Detrich, R. (2021). Overview of Professional Judgment. Oakland, CA: The Wing Institute. https://www.winginstitute.org/evidence-based-decision-making-professional-judgment.
August 31, 2021
Principal Evaluation. The field of principal evaluation, while gaining increased research interest in recent years, lags behind teacher evaluation in terms of conclusions that can be made regarding effective practice. Prior to Race to the Top and ESEA waivers, principal evaluation was implemented inconsistently and evaluation systems lacked instruments with validity and/or reliability, had a tenuous relationship with leadership standards, failed to include measures of student/school outcomes, and had mixed purposes as to their intended use (e.g., sometimes as formative information to help principals improve, while other times as summative information to make personnel decisions). However, today’s evaluation systems have evolved to incorporate multiple measures of principal performance that evaluate principals on research-based principles of effective leadership, often include student outcomes (which is often controversial, however), and are used both to help principals improve and to hold them accountable for their performance. Ongoing and more frequent observations, often conducted by the principal supervisor, who often also serves as a coach/mentor and directs the principal towards needed professional learning, show promise as an effective practice. Using the results from principal evaluations for personnel decisions, such as offering incentives through pay-for-performance programs, yields mixed results and warrants further research attention.
Citation: Donley, J., Detrich, R., States, J., & Keyworth, (2021). Principal Evaluation Oakland, CA: The Wing Institute. https://www.winginstitute.org/quality-leadership-principal-evaluation
August 25, 2021
Effective teacher retention bonuses: Evidence from Tennessee. The data are clear that students in high poverty schools perform worse on most measures of educational attainment; however, the discrepancy between high poverty schools and more affluent schools is reduced when there are quality teachers in the high poverty schools. The challenge is that teachers leave these schools at a higher rate. This turnover contributes to the poor outcomes for students in high poverty schools. Recruiting and training replacement teachers is an expensive proposition for districts. One approach to increasing retention in high poverty schools is to offer retention bonuses to teachers in these schools. There are two questions with respect to the use of retention bonuses: 1) are they effective over the long term, and 2) does having a more stable teachering corps increase student outcomes? A recent report examined the impact of teacher retention bonuses in Tennessee (Springer, Swain, & Rodriguez, 2016). The main findings are that teachers that participated in the retention bonus program were significantly more likely to stay in their school than teachers who did not participate. Importantly, the students in the classrooms of participating teachers had significantly higher academic gains than students of non-participating teachers. Looking at these data through the Active Implementation Frameworks lens, the retention bonus represents a usable innovation. The teacher retention bonuses are also an element of the Competency driver, specifically Selection. Finally, this innovation links to the Organizational driver since to effectively implement it, the innovation has to be considered a system level intervention.
Citation: Springer, M. G., Swain, W. A., & Rodriguez, L. A. (2016). Effective teacher retention bonuses: Evidence from Tennessee. Educational Evaluation and Policy Analysis, 38(2), 199-221.
August 25, 2021
Driven by Data: Using Licensure Tests to Build a Strong, Diverse Teacher Workforce. Essential to improving educational outcomes for students is to assure that well prepared teachers are in every classroom. Teacher preparation programs are primarily responsible for preparing candidates. One measure of how well institutions are preparing teachers is the percentage of candidates that pass state licensure tests. The National Council on Teacher Quality (https://www.nctq.org/) recently released a report examining the pass rate of elementary education teachers by state, by de-identified teacher preparation institutions, and disaggregated data for candidates of color and socio-economic status. Different states have different standards, rely on different methods to assess performance, and have different criteria for passing scores. Thirty-four states provided complete data for this report, eight provided partial data, and nine states provided no data. Based on the available data, nationally 55% of teacher candidates failed the exam on their first try. The data vary considerably across states and across institutions within and across states. One of the conclusions of this report is that elementary teacher candidates, regardless of race and ethnicity, are “too often poorly prepared and supported to pass their state licensure tests.” The authors of the report identified a number of issues with how states are currently assessing teacher competency. The report concludes with a number of recommendations for improving teacher preparation programs so that more teachers pass the licensure test. These data are directly relevant to the competency implementation driver in the Active Implementation Frameworks. Implementation efforts are not likely to be successful if competent personnel are not available to implement the innovation. Competency is primarily the responsibility of the teacher preparation programs. These programs would be well served to attend to the recommendations of this report. In addition, education policy makers should review their state’s current methods for assessing the competency of teacher candidates.
Citation: Putman, H. & Walsh, K. (2021). Driven by Data: Using Licensure Tests to Build a Strong, Diverse Teacher Workforce. Washington, D.C.: National Council on Teacher Quality
August 25, 2021
Principals’ Perceptions of Influence Over Decisions at Their Schools in 2017-2018. Contemporary models of principal leadership are that principals are expected to be the instructional leaders in their schools. At least two questions emerge from this expectation: (1) Do principals have the ability to influence instructional decisions in their schools? (2) Do principals have the necessary training to base instructional decisions on the best available evidence? In a recent report published by the National Center for Education Statistics at IES (July, 2021), the degree to which principals in traditional public schools, private schools, and charter schools felt like they have influence over decisions across a number of domains of school leadership was assessed. The degree to which they felt they had influence was related to the type of school in which they were working. Particularly, interesting from the perspective of principals as instructional leaders, is that only 39% of principals in traditional public schools felt like they had influence over establishing curriculum. This figure is considerably lower than for principals in private schools (69%) and principals in public charter schools (59%). This raises the question do traditional public school principals have a commitment to the curriculum choices that are made? If they do not, then one has to wonder if they will be champions for the curriculum and effective instructional leaders? In terms of the Active Implementation Frameworks, effective implementation of the model of principals as instructional leaders requires that principals be involved in the identification of useable innovations, the actual implementation of the innovation in their school, and access to the data about the effectiveness of the innovation. In addition, if principals are to be effective instructional leaders then the competency drivers of selection, training, and coaching need to be present so principals will have the necessary skills to function in those roles. Finally, the leadership drivers of technical skills and adaptive leadership skills are necessary to adapt an instructional practice into a particular organizational and school context. It would be interesting to see how the practices in private schools and public charter schools differ from traditional public schools that results in principals reporting they have influence over decisions involving curriculum. This article only addresses the question of do principals have influence over curriculum decisions? It does not address the extent to which principals have the skills to base decisions on best available evidence. Please see the Wing Institute paper on Best Available Evidence (https://www.winginstitute.org/evidence-based-decision-making-evidence) for more on this topic.
Citation: National Center for Education Statistics at IES. (2021). Principals’ Perceptions of Influence Over Decisions at Their Schools in 2017-2018.
August 25, 2021
Disproportionality reduction in exclusionary school discipline: A best-evidence synthesis. Disproportionality in the application of school discipline policies has been well documented over the years (Skiba, et al. (2011), and has been resistant to change. In a systematic review of the evidence of the effectiveness of programs and practices to reduce disproportionality, Cruz, Firestone, and Rodl (2021) found that the positive effects of individual programs such as School Wide Interventions and Support, and Restorative Justice, were either mixed or not evident. More promising were results from combining practices from different programs and including a specific equity framework for training educators. The most promising results were obtained when in-class coaching was a component of the training approach. The challenge of implementing a coaching model was identifying the necessary human and time resources, as well as the financial resources, to effectively implement coaching. This paper touches on several important aspects of the Active Implementation Frameworks including the identification of effective practices (usable innovations) and implementation drivers (professional development, leadership, and enabling contexts). The absence of any one of these features will limit the overall impact on efforts to reduce disproportionality.
Citation: Cruz, R. A., Firestone, A. R., & Rodl, J. E. (2021). Disproportionality reduction in exclusionary school discipline: A best-evidence synthesis. Review of Educational Research, 91(3), 397-431.