Educators are always looking for instructional methods that are effective and efficient. Effective interventions can vary with respect to how rapidly content is learned. Efficient methods result in rapid learning of content. Part of determining learning is establishing a mastery criterion (i.e., 90% correct over a specified number of days). The most common method for determining mastery is to establish a mastery criterion for a set of instructional content (i.e., sight words, math facts). Mastery is assumed when the percent correct score is at or above the mastery level (i.e., 90% correct). This approach may obscure that some items in the set have not been mastered, but the aggregate score is at mastery. Another way to determine mastery is to calculate it at the level of the individual item (individual sight words). Once an item is mastered, it is removed from the list, and a new item is added. The question is which approach results in greater learning? A recent study by Wong and Fienup (2022) was designed to answer the question, at least for sight words. Their results suggest that the individual item approach resulted in greater acquisition and required less time to achieve mastery of an item. An additional analysis in this small study was to compare the retention of items four weeks following the end of teaching. There were very small differences between the two approaches to instruction. For one participant, maintenance was 100% for both approaches. For a second participant, the individual item approach resulted in better maintenance scores. For the third participant, the set approach produced a slightly higher maintenance score.
The results of this study are important in that they suggest that the commonly used set approach is less efficient at producing initial acquisition and has no advantage with respect to the maintenance of mastered items. Implementing the individual item approach could be relatively simple. The only real change would be to analyze responding at the level of the individual item rather than aggregating data at the set level. As the student progresses through additional set lists and more difficult items are added, the student’s failure to have mastered all of the content may lead to more errors and failure experiences. If we can accelerate learning by making mastery decisions at the individual item level, consider how much more can be learned over the course of a school year. These simple changes may result in great benefit to students.
Wong, K. K., & Fienup, D. M. (2022). Units of analysis in acquisition‐performance criteria for “mastery”: A systematic replication. Journal of Applied Behavior Analysis.
Teacher evaluation is ubiquitous in US public schools. Typically, it involves a principal observing a teacher several times over the course of a school year. In an effort to standardize ratings, a scoring rubric is followed; however, the ratings are ultimately subjective, and the items on the rubric are subject to interpretation. One of the primary functions of teacher evaluations is to provide accurate feedback to teachers and encourage improvement when needed. A persistent question regarding teacher evaluation is if evaluation scores are inflated? There is some research suggesting this is the case; however, little is known about the reasons for inflating the scores. A recent study by Jones, Bergin, and Murphy (2022) attempted to determine if principals inflated scores and, if so, their motivation for doing so. Using a mixed method approach that utilized both focus groups and a survey of a large group of principals, principals identified several goals in addition to providing accurate ratings. Those additional goals were to (1) keep teachers open to growth-promoting feedback, (2) support teachers’ morale and foster positive relationships, (3) avoid difficult conversations, (4) maintain self-efficacy as an instructional leader, and (5) manage limited time wisely. These additional goals were offered as reasons to inflate scores, even if by small amounts. For the most part, these are worthy goals and suggest that teacher evaluation is more complicated than simply applying a scoring rubric while observing a teacher.
In general, principals are more likely to inflate ratings if they are linked to high-stakes outcomes such as requiring an improvement plan for the teacher or making retention decisions. Principals are reluctant to give lower ratings if it results in them having to engage in activities that require more time, such as additional meetings to develop improvement plans or to carefully document the reasons for recommending against retention. Also, by inflating ratings, principals avoid having difficult conversations with a teacher.
The principals’ worry was that if they gave a lower rating, teachers would become defensive and less open to feedback and growth. They also feared that low ratings would lover staff morale and positive relationships would be harmed. These concerns are not without merit. On a rating scale that ranges from 1-7, a rating of 4 is considered a low rating by the teacher, but a 5 is considered acceptable. The difference of one point is considered small by the principal. Since there is room for judgment in the scoring rubric giving a more positive rating will do no harm from the principal’s perspective.
Based on the research by Jones, Bergin, and Murphy (2022), these situations are relatively common. Overlooked in the principals’ perspective is that there is little consideration given to the impact these decisions have on students. It is unknown what effect these decisions are having on student outcomes. For a complete understanding of the evaluation of teachers, it is important to understand all of the effects of evaluations of teachers.
Citation for article:
Jones, E., Bergin, C., & Murphy, B. (2022). Principals may inflate teacher evaluation scores to achieve important goals. Educational Assessment, Evaluation, and Accountability, 34(1), 57-88.
Teachers report that behavior management is one of the greatest challenges in the profession and they feel unprepared to deal with difficult behavior. One of the questions to be answered is where do teachers get information about behavior management? Recently, Beahm, Yan, and Cook (2021) conducted a mixed methods study to answer this question. It is important that teachers rely on practices that have a good empirical base. Failure to do so may have no effect or make the problem worse. If we understand the resources teachers rely on and why, then more systematic, informed approaches can be taken to assure they are relying on credible information. This may help us close the research-to-practice gap. Beahm et al. surveyed 238 teachers to learn about the resources they relied on for behavior management information. They also did focus groups with 10 of the teachers to gain insight into why they preferred some resources more than others. Teachers preferred getting information from colleagues by a large margin (91%) relative to any other source, including research articles, the internet, administrators, and professional development. Ninety-two percent reported the information from colleagues was understandable. Teachers had a positive perception of all attributes of the information from colleagues (trustworthiness, usability, accessibility, and understandability). Participants in the focus group reported that colleagues were understandable because they used familiar language and avoided jargon. In addition, colleagues were perceived to provide exact details on implementing the recommended practice.
Participants in the focus group indicated colleagues were more trustworthy because they were going to only describe practices they had used successfully. The participants also thought that colleagues had knowledge of their classrooms and students.
Finally, colleagues were perceived as providing information that was usable because they likely had developed easy-to-use forms and data collection systems. In other words, the colleagues were an efficient source of information, saving the classroom teacher from the extra work of developing forms and data sheets for themselves.
These data are consistent with the recommendations of Rogers (2003), who reported that practices were more likely to be adopted if they were recommended by a credible source. Colleagues use language that is already familiar and have in-depth knowledge of the circumstances that the teacher is concerned with.
Researchers will be well served to attend to these data if they want to close the research-to-practice gap. They should develop materials that rely on the language teachers already use, create step-by-step user guides, and provide video samples of the practice in actual application. Finally, researchers should recruit teachers to be champions for a research-based practice rather than relying on researchers to disseminate practices. This would represent a change in the way researchers go about doing business. It will be worth the effort because the research-to-practice gap has been persistent for decades. It is time we try new ways to disseminate effective practices.
Beahm, L. A., Yan, X., & Cook, B. G. (2021). Where Do Teachers Go for Behavior Management Strategies? Education and Treatment of Children, 44(3), 201-213.
Link to article:
References: Rogers, E. M. (2003). Diffusion of Innovations (5th ed.). New York: Free Press
The measurement of treatment integrity is important any time an intervention is implemented. The measurement of treatment integrity is complex when assessing it at the level of universal intervention for an entire school. Should we measure integrity at the level of the school or at the level of the individual classroom? When assessed at the level of the whole school, we know how the school, in general, is performing; however, this may obscure how well an individual classroom is implementing the universal intervention. Assessment at the level of the classroom is important for making decisions regarding an individual student’s need for more intensive interventions (Tier 2 or Tier 3). If the universal intervention has not been implemented well, then it is difficult to know if the student’s failure to perform is a function of poor implementation or if the student requires more intensive support.
Given the importance of decision-making in multi-tiered systems of support, little is known about how integrity is measured. In a recent study by Buckman et al. (2021), treatment integrity of universal interventions was mapped in terms of frequency of measuring, the method used to assess, and unit of analysis (whole school or individual classroom). A systematic review of the published literature since 1996 resulted in 42 articles being included in this review. Over 86% of the articles reported procedures for monitoring integrity, and 76% reported quantifiable data for Tier 1 treatment integrity. These are encouraging data. The most common method for assessing treatment integrity was self-report (90%). Self-report measures are efficient, but there is the risk of the reports being inflated over what actually occurred. It is easy to understand why self-report is utilized so commonly, given the resource demands associated with measuring integrity across an entire system; however, much more research needs to be done to establish conditions for the self-reports to be valid measures. Direct observation was used least often to assess treatment integrity (18.75%). The resource demands make it very difficult to use even though it is most likely to yield the most valid data. Procedures to balance the efficiency and effectiveness of different methods for assessing integrity have yet to be fully developed.
Monitoring of treatment integrity occurred 81% of the time at the school level. 40% of the studies assessed treatment integrity at the level of the individual classroom. These measures are not mutually exclusive. In some instances, integrity was measured at both levels. Of the studies reviewed, 57% measured integrity one time per year. This raises questions about the representativeness of the data, especially when the data were most often collected at the level of the entire school. School-wide measurement obscures implementation at the classroom level, and measuring only one time per year may further obscure variables that influence the obtained data point. There is no established standard for the frequency of measuring integrity at the universal level of intervention. It could be argued that these measures should be employed at the same frequency at which decisions are made regarding students’ need for additional services. For example, if school-wide data are reviewed three times per year, then integrity measures should occur three times per year. This would allow decision-makers to track changes in integrity across time and determine if student performance reflects changes in integrity. All of this is done to increase the validity of decisions regarding the level of support required for individual students.
There are challenges to assessing integrity at the universal level. Considerable resources are required to assess across an entire school, especially when measuring at the level of the individual classroom. Efficient and effective systems that can be employed by existing school resources are necessary and have yet to be developed. The importance of these systems cannot be overstated. High-stakes decisions about students’ futures are being made based on their performance at the universal level of instruction. It is essential that the decisions are based on valid data, including treatment integrity data.
Citation: Buckman, M. M., Lane, K. L., Common, E. A., Royer, D. J., Oakes, W. P., Allen, G. E., … & Brunsting, N. C. (2021). Treatment integrity of primary (tier 1) prevention efforts in tiered systems: Mapping the literature. Education and Treatment of Children, 44(3), 145-168.
Evidence-based interventions have the potential to improve educational outcomes for students. Often these programs are introduced with an initial training but once the training has been completed often there is no additional follow-up support available. This can result in the educational initiative not being fully adopted and frequently abandoned soon after initial adoption. To change this cycle, on-going coaching or implementation support has been suggested as an alternative. The current study by Owen and colleagues evaluated the impact of implementation supports on student outcomes who participated in the implementation of Say All Fast Minute Every Day Shuffled (SAFMEDS). This program is designed to promote fast and accurate recall. In this instance, the goal was to increase fluency with math facts. This was a large randomized trial in which teachers received training on implementing SAFMEDS, and following training were assigned to either a No Support group, or an Implementation Support Group. Implementation Support consisted of three face-to-face meetings with a teacher and email contact initiated by the teacher. All of the students in the study had been identified as performing below standards for their age. The results suggest that across grade levels (Grade 1-2 and Grades 3-5) Implementation Supports resulted in small effect size improvements compared to the No Support Group. For Grades 1-2, the effect size was d=0.23 and for Grades 3-5 d=0.25. These are relatively small effect sizes; however, they are larger than the average effect sizes reported in the professional development literature that apply coaching elements to math programs. It should also be noted that the Implementation Supports consisted of three hours across a school year. This is a relatively low intensity dose of support and one that is likely to be practical in most school contexts.
The important take-away from this research is that some level of Implementation Support will likely be necessary to gain benefit from empirically-supported interventions such as SAFMEDS. The challenge for researchers is to identify the minimum dosage of Implementation Support to improve outcomes and the critical components of the Implementation Support so that it is efficient and effective.
Citation: Owen, K. L., Hunter, S. H., Watkins, R. C., Payne, J. S., Bailey, T., Gray, C., … & Hughes, J. C. (2021). Implementation Support Improves Outcomes of a Fluency-Based Mathematics Strategy: A Cluster-Randomized Controlled Trial. Journal of Research on Educational Effectiveness, 14(3), 523-542.
Performance feedback is often considered a necessary part of training educators. the challenge is to provide the feedback in a timely manner so that it positively impacts skill acquisition. Often times, the feedback is delayed by hours, or days, which may limit the impact of the feedback. Real-time performance feedback is considered optimal, but may be considered unfeasible in many educational contexts.
One option is to provide feedback utilizing technology such as “bug in the ear” to deliver feedback in real-time. Sinclair and colleagues (2020) conducted a meta-analysis to determine if feedback delivered via technology could be considered to empirically-supported. In the review, 23 studies met inclusion criteria. Twenty-two of the studies were single case designs and one was a group design. The reported findings were that real-time performance feedback is an effective method for increasing skill acquisition of educators. The authors cautioned that this type of feedback is an intensive intervention and suggested that it is not feasible to use for training all teachers. They suggest that it should be considered an intervention when other training methods have not proven effective.
In this context, it becomes feasible to support those educators that have not benefitted from less intensive interventions. If it is considered part of a multi-tiered system of support for educators, it can play an important role in training. It can improve the performance of educators and perhaps reduce turnover because it allows educators to develop the skills to be successful.
Sinclair, A. C., Gesel, S. A., LeJeune, L. M., & Lemons, C. J. (2020). A review of the evidence for real-time performance feedback to improve instructional practice. The Journal of Special Education, 54(2), 90-100.
Classroom teachers consistently report classroom management as a significant area of concern. This is especially true for early career teachers and teachers often report it is one of the most common reasons for leaving the profession. Highly rigorous, practical, and effective pre-service and professional development training approaches are necessary to address classroom behavior challenges. A recent systematic review by Hirsch and colleagues (2021) reviewed the literature on classroom management training to determine the current status of professional development for classroom teachers. Ultimately, the authors identified eight experimental studies that met inclusion criteria. There were several interesting findings from this review. Of the experimental studies reviewed, a low number of participants reported having received prior training in classroom management. As the authors discuss, these results are not surprising since relatively few states have policy requirements for classroom teachers to receive instruction in classroom management. Stevenson and colleagues (2020) proposed the steps for improving instruction in classroom management: (1) pre-service coursework must include a course on explicit, evidence-based, culturally, and contextually relevant classroom management skills; (2) fieldwork should incorporate explicit support and coaching on classroom management; and (3) state departments of education should require training that aligns with the best practices of classroom management to support the needs of teachers and students. If these three recommendations were acted on, teachers would likely be more prepared to address the behavioral challenges in their classrooms.
A second finding from the Hirsch et al. (2021) systematic review was that there is considerable evidence to support practice-based professional development rather than the standard “train and hope” (Stokes & Baer, 1977). There are seven critical features to practice-based professional development. In the articles reviewed in this systematic review, all of the studies incorporated some elements of practice-based professional development. A somewhat surprising finding among the reviewed articles was that the length of training ranged from 15 minutes to four days. This result is likely possible because the researchers used practice-based professional development that included coaching and feedback to teach the new skills.
Hirsch and colleagues made a strong argument for the increased use of technology to support professional development, ranging from low-tech methods to telehealth. Telehealth makes it possible for teachers in rural communities to access high-quality professional development. Creating more effective and efficient professional development is necessary to scale it up.
As Hirsch et al. make clear, considerably more research on professional development is necessary. Eight articles are a small database for making policy and practice recommendations.
Hirsch, S. E., Randall, K., Bradshaw, C., & Lloyd, J. W. (2021). Professional Learning and Development in Classroom Management for Novice Teachers: A Systematic Review. Education and Treatment of Children, 44(4), 291-307.
Stevenson, N. A., VanLone, J., & Barber, B. R. (2020). A commentary on the misalignment of teacher education and the need for classroom behavior management skills. Education and Treatment of Children, 43(4), 393-404.
Stokes, T. F., & Baer, D. M. (1977). An implicit technology of generalization 1. Journal of applied behavior analysis, 10(2), 349-367.
One of the great challenges in education is training all staff to implement interventions. There is considerable reliance on para-professionals, especially in special education, to support students. Many of the para-professionals have minimal training in educational practices. In many cases, the training that does occur is the traditional didactic model and there is little evidence that it produces the outcomes it is supposed to yield. An alternative model of training that holds great promise is coaching; however, there are limitations to it because it often relies on outside coaches which makes it cost-prohibited for many districts. A recent report by Sallese and Vannest (2021) offers an alternative that may make coaching more cost-effective. In their research, they utilized classroom teachers to coach the para-professionals working in the classroom to increase the use of behavior specific praise. Many teachers report that they have little or no pre-service or in-service training focused on paraprofessional training and support (Douglas, Chapin & Nolan, 2016). To address this issue, the teachers were provided a manual to guide their coaching efforts. The components of the coaching package included self-monitoring, performance feedback, goal setting, modeling, and action planning. In surveys of paraprofessionals one of the most cited concerns is lack of training and support in behavior management (Mason, et al., 2021). Behavior specific praise has been identified as an evidence-based component of classroom behavior management (Simonsen, Fairbanks, Briesch, Myers, & Sugai (2008); however, it has often been a challenge to increase behavior specific praise and maintain it over time. In this study, all four of the para-professionals that received coaching increased their rate of behavior specific praise. In addition, 100% of the participants agreed that the procedures were appropriate and feasible in terms of time and effort required to implement.
This was a small-scale study but it holds promise as a method for coaching implementers to carry out effective practices. From an implementation perspective, this provides a cost-effective approach to increase the internal capacity of a system to implement adopted practices. Building internal capacity is critical if effective interventions are to be sustained over generations of implementers.
Citation: Sallese, M. R., & Vannest, K. J. (2021). Effects of a Manualized Teacher-Led Coaching Intervention on Paraprofessional Use of Behavior-Specific Praise. Remedial and Special Education, 07419325211017298.
Douglas, S. N., Chapin, S. E., & Nolan, J. F. (2016). Special education teachers’ experiences supporting and supervising paraeducators: Implications for special and general education settings. Teacher Education and Special Education, 39(1), 60–74. https://doi.org/gf86tz
Mason, R. A., Gunersel, A. B., Irvin, D. W., Wills, H. P., Gregori, E., An, Z. G., & Ingram, P. B. (2021). From the frontlines: Perceptions of paraprofessionals’ roles and responsibilities. Teacher Education and Special Education, 44(2), 97–116. https://doi.org/fwn6
Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). Evidence-based practices in classroom management: Considerations for research to practice. Education and treatment of children, 351-380.
Teacher Preparation Program Models Overview. Teacher preparation began in the mid-19th century with the normal school, a 2-year course of study that prepared candidates for teaching. This model remained unchanged until the early 20th century, when universities created the undergraduate model, which currently predominates. Teacher candidates are required to spend 4 years obtaining a bachelor’s degree built around a prescribed course of education study. A second relatively recent modification is the 5-year credential model, requiring candidates to obtain a bachelor’s degree before beginning a 5th year of instruction in teaching. The driving force behind the postgraduate model was the belief that teachers were not respected. It was assumed that a post-bachelor’s and/or graduate degree certificate would confer greater esteem on the profession. This model is offered across the country and is mandated for all new teachers in California. A third option, the alternative credential (AC) model, arose as a solution to teacher shortages. The AC model is distinct from the traditional models in that candidates receive formal preparation coursework while already employed in the classroom. Currently, little evidence exists to support the superiority of any one method over the others.
Teacher Preparation: Instructional Effectiveness Overview. Discussions of teacher preparation generally focus on content (what to teach) rather than pedagogy (how to teach). Teacher training has changed little in 100 years. Preparation programs rely on lectures supplemented with 8 weeks of student teaching under minimal university oversight. Lecturing persists for various reasons: It requires nominal effort, instructors have greater control of what is presented, and assessing mastery of the material is easy using tests and papers. There are significant disadvantages to lecturing. Listening to a lecturer and answering questions during the lecture are very different from being able to perform skillfully in a real-world setting. Research shows that the most effective training of complex skills occurs when the training follows the elementary paradigm “I do,” “we do,” “you do.” This model relies on introducing skills through lectures and discussions, in tandem with demonstrating the skills (I do). This is followed by learners practicing the skills alongside a coach (we do), and finally the student teacher performing independently with feedback from the coach (you do). Research suggests it is only when coaching is added to the mix that skills are fully mastered and used effectively in the classroom.
Citation: Cleaver, S., Detrich, R., States, J. & Keyworth, R. (2021). Teacher Preparation: Instructional Effectiveness. Oakland, CA: The Wing Institute. https://www.winginstitute.org/pre-service-teacher-instructional-effectiveness.