Education Drivers

Value Added

Value-added modeling (VAM) is a statistical approach that provides quantitative performance measures for monitoring and evaluating schools and other aspects of the education system. VAM comprises a collection of complex statistical techniques that use standardized test scores to estimate the effects of individual schools or teachers on student performance. 

Value-added modeling (VAM) is a statistical approach that provides quantitative performance measures for monitoring and evaluating schools and other aspects of the education system. VAM comprises a collection of complex statistical techniques that use standardized test scores to estimate the effects of individual schools or teachers on student performance. Although the VAM approach holds promise, serious technical issues have been raised regarding VAM as a high-stakes instrument in accountability initiatives. The key question remains: Can VAM scores of standardized test scores serve as a proxy for measuring teaching quality? To date, research on the efficacy of VAM is mixed. There is a body of research that supports VAM, but there is also a body of studies suggesting that model estimates are unstable over time and subject to bias and imprecision. A second issue with VAM is the sole use of standardized tests as a measure of student performance. Despite these valid concerns, VAM has been shown to be valuable in performance improvement efforts when used cautiously in combination with other measures of student performance such as end-of-course tests, final grades, and structured classroom observations.

Data Mining

TITLE
SYNOPSIS
CITATION
What is the relationship between teacher working conditions and school performance?
This inquiry looks at the effect of time on the job and the quality of a teacher's skills.
Keyworth, R. (2010). What is the relationship between teacher working conditions and school performance? Retrieved from what-is-relationship-between882.
TITLE
SYNOPSIS
CITATION
ASA statement on using value-added models for educational assessment

Value-Added Models (VAMs) has been embraced by many states and school districts as part of educational accountability systems. Value-Added Assessment (VAA) Models attempt to estimate effects of individual teachers or schools on student achievement while accounting for differences in student background. This paper provides a summary of the American Statistical Associations analysis of the efficacy of value-added modeling in education.

American Statistical Association. (2014). ASA statement on using value-added models for educational assessment. Alexandria, VA.

The Value of Interrupted Time-Series Experiments for Community Intervention Research

This paper advocates the use of time-series experiments for the development and evaluation of community interventions.

Biglan, A., Ary, D., & Wagenaar, A. C. (2000). The value of interrupted time-series experiments for community intervention research. Prevention Science1(1), 31-49.

Teacher Preparation and Student Achievement.

This article examined the differences in effectiveness of teacher preparation programs that supply teachers to New York City schools.  One of the important findings is that preparation directly linked to practice benefits teachers in their first year.

Boyd, D. J., Grossman, P. L., Lankford, H., Loeb, S., & Wyckoff, J. (2009). Teacher Preparation and Student Achievement. Educational Evaluation & Policy Analysis, 31(4), 416-440.

Using Student Progress to Evaluate Teachers: A Primer on Value-Added Models. Policy Information Perspective.

This report is a lay person’s guide to value added modeling as a means of evaluating teacher performace.

Braun, H. I. (2005). Using Student Progress to Evaluate Teachers: A Primer on Value-Added Models. Policy Information Perspective. Educational Testing Service. Retrieved from http://files.eric.ed.gov/fulltext/ED529977.pdf

 

The debate about rewards and intrinsic motivation: Protests and accusations do not alter the results.

A prevailing view in education and social psychology is that rewards decrease a person’s intrinsic motivation. However, our meta-analysis (Cameron & Pierce, 1994) of approximately 100 studies does not support this position. The only negative effect of reward occurs under a highly restricted set of conditions, circumstances that are easily avoided. These results have not been well received by those who argue that rewards produce negative effects under a wide range of conditions. Lepper, Keavney, and Drake (1996)Ryan and Deci (1996), and Kohn (1996) have suggested that the questions asked in our meta-analysis were inappropriate, that critical studies were excluded, that important negative effects were not detected, and that the techniques used in our meta-analysis were unsuitable. In this response, we show that the questions we asked are fundamental and that our meta-analytic techniques are appropriate, robust, and statistically correct. In sum, the results and conclusions of our meta-analysis are not altered by our critics’ protests and accusations.

Cameron, J., & Pierce, W. D. (1996). The debate about rewards and intrinsic motivation: Protests and accusations do not alter the results. Review of Educational Research, 66(1), 39–51.

The long-term impacts of teachers: Teacher value-added and student outcomes in adulthood

Are teachers' impacts on students' test scores ("value-added") a good measure of their quality? This question has sparked debate largely because of disagreement about (1) whether value-added (VA) provides unbiased estimates of teachers' impacts on student achievement and (2) whether high-VA teachers improve students' long-term outcomes. We address these two issues by analyzing school district data from grades 3-8 for 2.5 million children linked to tax records on parent characteristics and adult outcomes. We find no evidence of bias in VA estimates using previously unobserved parent characteristics and a quasi-experimental research design based on changes in teaching staff. Students assigned to high-VA teachers are more likely to attend college, attend higher- ranked colleges, earn higher salaries, live in higher SES neighborhoods, and save more for retirement. They are also less likely to have children as teenagers. Teachers have large impacts in all grades from 4 to 8. On average, a one standard deviation improvement in teacher VA in a single grade raises earnings by about 1% at age 28. Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students' lifetime income by more than $250,000 for the average class- room in our sample. We conclude that good teachers create substantial economic value and that test score impacts are helpful in identifying such teachers.

 

Chetty, R., Freidman, J. N., & Rockhoff, J. E. (2011). The long-term impacts of teachers: Teacher value-added and student outcomes in adulthood. (Working Paper 17699). Cambridge, MA: National Bureau of Economic Research. Every Student Succeeds Act, 20 U.S.C. § 6301 et seq. (2015).

The Long-Term Impacts Of Teachers: Teacher Value-Added And Student Outcomes In Adulthood

This paper examines the issue of efficacy of value-added measures in evaluating teachers. This question is important in understanding whether value-added analysis provides unbiased estimates of teachers’ impact on student achievement and whether these teachers improve long-term student outcomes.

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2011). The long-term impacts of teachers: Teacher value-added and student outcomes in adulthood (No. w17699). National Bureau of Economic Research.

Sharing successes and hiding failures: ‘reporting bias’ in learning and teaching research

This paper examines factors that lead to bias as well offers specific recommendations to journals, funders, ethics committees, and universities designed to reduce reporting bias.

Dawson, P., & Dawson, S. L. (2018). Sharing successes and hiding failures:‘reporting bias’ in learning and teaching research. Studies in Higher Education43(8), 1405-1416.

The Effect of Career and Technical Education on Human Capital Accumulation: Causal Evidence from Massachusetts

Twenty percent of high school students take four or more courses in career and technical education (CTE). Despite this high rate of participation, little is known about what constitutes high-quality CTE and whether high-quality CTE allows participants to accumulate meaningful knowledge and skills to succeed in a career. This study from the Association for Education Finance and Policy examined the impact of participating in CTE on high school attendance, high school completion, professional certifications, and performance on standardized test scores. The evidence suggests that a high-quality CTE program boosts on-time graduation for higher income students and for lower income .

 

Dougherty, S. M. (2016). The effect of career and technical education on human capital accumulation: Causal evidence from Massachusetts. Education Finance and Policy. doi:10.1162/EDFP_a_00224.

Education Equality in America Comparing the Achievement Gap Across School and Cities

Education Cities and GreatSchools have together launched the Education Equality Index in an attempt to answer "how does the U.S. fare in our effort to provide equal opportunity to all children?" question. The Education Equality Index is the first national comparative measure of the achievement gap between children growing up in low-income communities and their more advantaged peers.

Education Equality in America Comparing the Achievement Gap Across School and Cities. (2016, March). Education Equality Index. Retrieved from http://www.educationequalityindex.org/wp-content/uploads/2016/02/Education-Equality-in-America-v1-4.pdf

Selecting growth measures for school and teacher evaluations: Should proportionality matter?

In this paper we take up the question of model choice and examine three competing approaches. The first approach, (SGPs) framework, eschews all controls for student covariates and schooling environments. The second approach, value-added models (VAMs), controls for student background characteristics and under some conditions can be used to identify the causal effects of schools and teachers. The third approach, also VAM-based, fully levels the playing field so that the correlation between school- and teacher-level growth measures and student demographics is essentially zero. We argue that the third approach is the most desirable for use in educational evaluation systems.

Ehlert, M., Koedel, C., Parsons, E., & Podgursky, M. (2013). Selecting growth measures for school and teacher evaluations: Should proportionality matter?. National Center for Analysis of Longitudinal Data in Education Research, 21.

Evidence for the need to more closely examine school effects in value-added modeling and related accountability policies

This paper evaluates the reasonableness of assumptions of weighted approaches to account for building level effects when value added modeling is used to evaluate teachers.  In urban schools and ultimately teachers in those schools were negatively impacted by using weighted models to account for building level effects.

Franco, M. S., & Seidel, K. (2014). Evidence for the need to more closely examine school effects in value-added modeling and related accountability policies. Education and Urban Society. Retrieved from http://journals.sagepub.com/doi/abs/10.1177/0013124511432306

The legal and policy implications of value-added teacher assessment policies

This argument makes the case that policies that require states’ student test scores to account for 40-50% of the overall score of teacher performance in value added modeling may produce a significant number of teachers to be falsly identified as ineffective.  Using such value added models may leave school districts legally vulnerable to law suits.

Green, P. C., Baker, B. D., & Oluwole, J. (2012). The legal and policy implications of value-added teacher assessment policies. BYU Educ. & LJ. Retrieved fromhttp://digitalcommons.law.byu.edu/elj/vol2012/iss1/2

Reliability and Validity of Inferences about Teachers Based on Student Scores

Policymakers and school administrators have embraced value-added models of teacher effectiveness as tools for educational improvement. Teacher value-added estimates may be viewed as complicated scores. This Paper examines the use of value-added modeling as a tool to identify effective teachers from ineffective instructors.

Haertel, E. H. (2013). Reliability and Validity of Inferences about Teachers Based on Student Scores. William H. Angoff Memorial Lecture Series. Educational Testing Service.

A Review of value-added models

This paper examines issues of value-added modeling. It describes what value-added modeling is and how it works in education.

Hibpshman, T. L. (2004). A Review of value-added models. Kentucky Education Professional Standards Board.

Estimating teacher impacts on student achievement: An experimental evaluation

This study used a random-assignment experiment in Los Angeles Unified School District to evaluate various non-experimental methods for estimating teacher effects on student test scores. Having estimated teacher effects during a pre-experimental period, the authors used these estimates to predict student achievement following random assignment of teachers to classrooms.

Kane, T. J., & Staiger, D. O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation (No. w14607). National Bureau of Economic Research.

Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment.

In this study the authors designed the Measures of Effective Teaching (MET) project to test replicable methods for identifying effective teachers. In past reports, the authors described three approaches to measuring different aspects of teaching: student surveys, classroom observations, and a teacher's track record of student achievement gains on state tests.

Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. Research Paper. MET Project. Bill & Melinda Gates Foundation.

Identifying effective classroom practices using student achievement data

This paper combines information from classroom-based observations and measures of teachers' ability to improve student achievement as a step toward addressing these challenges. The results point to the promise of teacher evaluation systems that would use information from both classroom observations and student test scores to identify effective teachers.

Kane, T. J., Taylor, E. S., Tyler, J. H., & Wooten, A. L. (2011). Identifying effective classroom practices using student achievement data. Journal of human Resources, 46(3), 587-613.

 

Value-added modeling: A review

This article provides a review of areas of agreement and disagreement of vaious aspects of value added modeling.

Koedel, C., Mihaly, K., & Rockoff, J. E. (2015). Value-added modeling: A review. Economics of Education Review. Retrieved from http://faculty.smu.edu/millimet/classes/eco7321/papers/koedel et al 2015.pdf

 

The politics and statistics of value-added modeling for accountability of teacher preparation programs

This paper reviews the evaluation of a value added model for evaluating teacher preparation programs in Texas.  The model produced statistically meaningful data but the results were sensitive to decisions about accountability critieria, the selection of teachers, and the selection of control variables.  Different decisions across each of these variables would impact the results of the value added modeling.

Lincove, J. A., Osborne, C., & Dillon…, A. (2014). The politics and statistics of value-added modeling for accountability of teacher preparation programs. … of Teacher Education. Retrieved from http://journals.sagepub.com/doi/abs/10.1177/0022487113504108

The Characteristics and Experiences of Youth in Special Education. Findings from the National Longitudinal Transition Study 2012. Volume 2: Comparisons across Disability Groups

The United States has committed to improving the lives of students with disabilities for over 40 years. Since the advent of Federal Law PL 94-142 in 1975 that mandated a free and appropriate education for all students regardless of ability and six reauthorizations of legislation, the federal government has emphasized the need to prepare students with disabilities for post-secondary education, careers, and independent living. The federal investment in funding special education services exceeds $15 Billion annually. It is reasonable to ask, are student with disabilities substantially benefiting from these efforts? The National Longitudinal Transition Study (NLTS) provides the most recent data on youth with disabilities and efforts to address their needs. The study used surveys in 2012 and 2013 on a nationally representative set of nearly 13,000 students. The student included were mostly those with an individualized education program (IEP) and expected to receive special education services. The data reveal participation in key transition activities by youth with an IEP and their parents have declined, although they are just as likely to have gone to an IEP meeting. The findings from this report suggest a closer examination of current practices is warranted with a focus on achieving the stated outcomes the laws were designed to remedy.

Lipscomb, S., Hamison, J., Liu Albert, Y., Burghardt, J., Johnson, D. R., & Thurlow, M. (2018). Preparing for Life after High School: The Characteristics and Experiences of Youth in Special Education. Findings from the National Longitudinal Transition Study 2012. Volume 2: Comparisons across Disability Groups. Full Report. NCEE 2017-4018. National Center for Education Evaluation and Regional Assistance.

Education Pays 2016: The Benefits of Higher Education for Individuals and Society.

This report documents differences in the earnings and employment patterns of U.S. adults with different levels of education. It also compares health-related behaviors, reliance on public assistance programs, civic participation, and indicators of the well-being of the next generation. This year's report also presents data on variation in earnings by different characteristics such as gender, race/ethnicity, occupation, college major, and sector. 

Ma, J., Pender, M., & Welch, M. (2016). Education Pays 2016: The Benefits of Higher Education for Individuals and Society. Trends in Higher Education Series. College Board.

The promise and peril of using value-added modeling to measure teacher effectiveness

This article addresses the potential sources of bias that can be introduced into value added modeling by the decisions that are made about the  details of the model.  There is a call for a refinement of procedures used when applying value added modeling.

McCaffrey, D. F., Koretz, D., Lockwood, J. R., & Hamilton, L. S. (2004). The promise and peril of using value-added modeling to measure teacher effectiveness. rand.org. Retrieved from http://www.rand.org/pubs/research_briefs/RB9050

Evaluating Value-Added Models for Teacher Accountability. Monograph.

Value added modeling has become of interest to policymakers interested in evaluating teacher performance.  The authors argue that the models work well when the schools in the sample are homogenous but as heterogeneity of the student population  increases estimates of teacher effects are likely to confounded.

McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating Value-Added Models for Teacher Accountability. Monograph. ERIC. Retrieved from http://eric.ed.gov/?id=ED529961

Models for value-added modeling of teacher effects

Value added modeling has become of interest to policymakers interested in evaluating teacher performance.  The authors argue that the models work well when the schools in the sample are homogenous but as heterogeneity of the student population  increases estimates of teacher effects are likely to confounded.

McCaffrey, D. F., Lockwood, J. R., Koretz, D., Louis, T. A., & Hamilton, L. (2004). Models for Value-Added Modeling of Teacher Effects. Journal of Educational and Behavioral Statistics, 29(1), 67-101. doi:10.3102/10769986029001067

Alternative student growth measures for teacher evaluation: Implementation experiences of early-adopting districts

This study examines implementation of alternative student growth measures in a sample of eight school districts that were early adopters of the measures. It builds on an earlier Region­ al Educational Laboratory Mid-Atlantic report that described the two types of alterna­tive student growth measures—alternative assessment–based value-added models and student learning objectives—in the early-adopting districts.

McCullough, M., English, B., Angus, M. H., & Gill, B. (2015). Alternative student growth measures for teacher evaluation: Implementation experiences of early-adopting districts (No. 8a9dfcb1bc6143608448114ea9b69d06). Mathematica Policy Research.

Where is the value in value-added modeling

This paper reviews the various issues and challenges associated with value added modeling.  It concludes with a discussion of how value added modeling  can be used in conjuction with other measures to identify teacher strengths and weaknesses.

Murphy, D. (2012). Where is the value in value-added modeling. 24–26 September 2014. Retrieved from http://educatoreffectiveness.pearsonassessments.com/downloads/viva_v1.pdf

Value-Added Assessment of Teacher Preparation: An Illustration of Emerging Technology

This paper describes one effort to use value added modeling to evaluate teacher preparation program in one state.

Noell, G. H., & Burns, J. L. (2006). Value-Added Assessment of Teacher Preparation: An Illustration of Emerging Technology. Journal of Teacher Education, 57(1), 37-50. doi:10.1177/0022487105284466

doi: 10.1177/0022487105284466

How teacher turnover harms student achievement

This study used a version of value added modeling to evaluate the impact of teacher turnover has on student achievement.

Ronfeldt, M., Lankford, H., Loeb, S., & Wyckoff, J. (2011). How Teacher Turnover Harms Student Achievement. National Bureau of Economic Research Working Paper Series, No. 17176. doi:10.3386/w17176

The Hidden Cost of California’s Harsh School Discipline: And the Localized Economic Benefits from Suspending Fewer High School Students

This research from the Center for Civil Rights Remedies at the Civil Rights Project, UCLA, and California Dropout Research Project shows that the overuse of suspensions in California schools is harming student achievement and graduation rates, and causing billions of dollars in economic damage. The financial consequences of school suspensions, including both additional costs borne by taxpayers as a result of suspensions and lost economic benefit, are quantified. The impact of school suspension varies widely by school district, with California’s largest districts incurring the greatest losses. For example, suspensions in the Los Angeles Unified School District for a 10th grade cohort are estimated to cause $148 million in economic damage. The report calculates a total statewide economic burden of $2.7 billion over the lifetime of the single 10th grade cohort.

Rumberger, R., & Losen, D. (2017). The Hidden Cost of California’s Harsh School Discipline: And the Localized Economic Benefits from Suspending Fewer High School Students. The Center for Civil Rights Remedies at the Civil Rights Project, UCLA, and California Dropout Research Project.

The Tennessee value-added assessment system (TVAAS): Mixed-model methodology in educational assessment

This paper describes the mixed method model Tennessee developed to evaluate teacher contributions to student achievement.  It desribes how they resolved some of the challenges to using value added modeling.

Sanders, W. L., & Horn, S. P. (1994). The Tennessee value-added assessment system (TVAAS): Mixed-model methodology in educational assessment. Journal of Personnel Evaluation in education. Retrieved from https://eric.ed.gov/?id=EJ498467

Cumulative and residual effects of teachers on future student academic achievement.

The Tennessee Value-Added Assessment System determines the effectiveness of school systems, schools, and teachers based on student academic growth over time. Research conducted utilizing data from the TVAAS database has shown that race, socioeconomic level, class size, and classroom heterogeneity are poor predictors of student academic growth. Rather, the effectiveness of the teacher is the major determinant of student academic progress.

Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement.

Measuring teaching using value-added modeling: The imperfect panacea

This paper attempts to untangle some of the competing claims about the value of value added modeling.  It concludes that it should not be used as the sole measure of teacher performance but should be part of a larger accountability system.

Scherrer, J. (2011). Measuring teaching using value-added modeling: The imperfect panacea. NASSP Bulletin. Retrieved from https://eric.ed.gov/?id=EJ938929.

The Impacts of Reading Recovery at Scale: Results From the 4-Year i3 External Evaluation

A recent large-scale evaluation of Reading Recovery, a supplemental reading program for young struggling readers, supports previous research that found it to be effective.  In a 4 year, federally funded project, almost 3,500 students in 685 schools found that generally students benefitted from the intervention. Students receiving Reading Recovery receive supplemental services in a 1:1 instructional setting for 30 minutes 5 days a week from an instructor trained in Reading Recovery.  In the study reported here, students who received Reading Recovery had effect sizes of .35-.37 relative to a control group across a number of measures of reading.  These represent moderate effect sizes and account for about a 1.5 month increase in skill relative to the control group.  Even though the research supports the efficacy of the intervention, it also raises questions about its efficiency.  The schools that participated in the study served about 5 students and the estimated cost per student has ranged from $2,000-$5,000.  These data raise questions about the wisdom of spending this much money per student for growth of about a month and a half.

Sirinides, P., Gray, A., & May, H. (2018). The Impacts of Reading Recovery at Scale: Results From the 4-Year i3 External Evaluation. Educational Evaluation and Policy Analysis, 0162373718764828.

Teacher job satisfaction and motivation to leave the teaching profession: Relations with school context, feeling of belonging, and emotional exhaustion.

This study examines the relations between school context variables and teachers’ feeling of belonging, emotional exhaustion, job satisfaction, and motivation to leave the teaching profession. Six aspects of the school context were measured: value consonance, supervisory support, relations with colleagues, relations with parents, time pressure, and discipline problems.

Skaalvik, E. M., & Skaalvik, S. (2011). Teacher job satisfaction and motivation to leave the teaching profession: Relations with school context, feeling of belonging, and emotional exhaustion. Teaching and teacher education27(6), 1029-1038.

Evaluating special educator effectiveness: Addressing issues inherent to value-added modeling

This paper addresses the unique challenges posed by special education when using value added modeling to evaluate teacher effectiveness.

Steinbrecher, T. D., Selig, J. P., Cosbey, J., & Thorstensen, B. I. (2014). Evaluating Special Educator Effectiveness. Exceptional Children, 80(3), 323-336. doi:10.1177/0014402914522425

 

A Meta-Analysis of Direct Instruction

A soon to be published meta-analysis of Direct Instruction (DI) curricula that reviews research on DI curricula between 1966-2016 reports that DI curricula produced moderate to large effect sizes across the curriculum areas reading, math, language, and spelling.  The review is notable because it reviews a much larger body of DI research than has occurred in the past and covers a wide range of experimental designs (from single subject to randomized trials).  328 studies were reviewed and almost 4,000 effects were considered.  Given the variability in research designs and the breadth of the effects considered, it suggests that DI curricula produce robust results.  There was very little decline during maintenance phases of the study and greater exposure to the curricula resulted in greater effects.

Stockard, J., Wood, T. W., Coughlin, C. & Khoury, C. R. (in press), Review of Educational Research.  DOI: 10.3102/0034654317751919

 

TITLE
SYNOPSIS
University of Pennsylvania: A Grand Bargain for Education Reform

This web site provides resources for educators to better under what value-added modeling is and how it works.

Back to Top