Definition

Effectiveness research is conducted to answer questions about the impact of an intervention as it is “taken to scale” and implemented in typical practice settings.

 Executive Summary

Effectiveness research is primarily concerned with answering questions about the impact of an intervention as it is implemented in typical practice settings. As an intervention is moved from research settings to practice settings there are several variables that can influence the overall impact of an intervention.

Typically, in effectiveness research the unit of analysis is populations of students, classrooms, or schools. As a result the impact of an intervention on an individual student is obscured. One of the primary concerns of effectiveness research is how broad an impact will an intervention have so the impact on an individual student is less important.

Because effectiveness research is concerned with the impact on a population or setting, there is a generalization from the research setting to a specific practice setting that is required. To the extent that the practice setting is similar to the research setting then we can have greater confidence in the impact of an intervention. Nonetheless it is a generalization and as such, it can be less than perfect.

Effectiveness research answers questions about the probability of impact with a particular population or in a specific setting if the intervention is implemented as prescribed but it does not answer questions about how to actually implement the intervention.

Effectiveness research is less common than efficacy research. The reason for this is that there are some institutional and logistical barriers that make effectiveness research more difficult.

After an intervention has been demonstrated to be promising, the function of effectiveness research is to determine how robust the intervention is when it is implemented in settings similar to typical practice settings. Among the dimensions that have to be considered as an intervention is taken to scale are:

1. Which populations are likely to respond to the intervention?
2. Which populations will not benefit?
3. What are the characteristics of the setting are necessary for an intervention to be effective?
4. What level of training is required for those implementing the intervention?
5. What level of treatment integrity is necessary to produce beneficial effects?
6. What “dose” of the intervention is necessary?
7. How acceptable is the intervention to those who are responsible for implementation?
8. How powerful are the effects of this intervention compared to other interventions?
9. What are the typical costs of implementing the intervention?
   

Effectiveness research is still research so the standards of scientific rigor are still required. Often the unit of analysis changes as we move from efficacy to effectiveness research. In efficacy research, the unit of analysis is often the performance on individual students. In effectiveness research the unit of analysis is at a larger scale such as populations of students, classrooms, schools, characteristics of staff, etc. The impact on an individual student is less clear in this level of analysis but the impact on the larger units is better understood. This change in the unit of analysis highlights the fundamental difference between efficacy and effectiveness research. Each type of research is designed to answer different questions.

Effectiveness research does not answer questions about how effective an intervention will be in a specific setting but it can guide decision makers to consider the characteristics of the population and the settings in which the research was conducted relative to the characteristics of the populations and settings in which the decision maker is concerned. The more similar the research setting and the practice setting, the greater confidence we can have that the intervention will produce positive effects; however, it is still a generalization from the research to practice settings so there will be occasions in which an effective intervention will not result in positive outcomes in a particular practice setting. Generalizations from one context to another are often less than perfect.

Effectiveness research does not answer how to accomplish implementation within a specific setting. Effectiveness research only identifies interventions that are likely to produce positive outcomes. Questions of how to implement an intervention are answered by other lines of research. The transition from research to practice is one of the biggest challenges facing those interested in developing a culture of evidence-based decision-making within education.

Relative to efficacy research there is little effectiveness research being conducted. There are several reasons for this:

1. Effectiveness research generally requires greater numbers of participants than efficacy research. The greater numbers are required to establish that an intervention is generally effective with a population. Recruiting greater number of participants and actually exposing them to the experimental intervention generally requires more time and effort than is required of efficacy research.
2. University professors operating under the rules of promotion and tenure conduct most research. Generally, these rules require a high level of productivity as measured by publication in professional journals and effectiveness research often takes more time per study than efficacy research.
3. The rules for promotion and tenure encourage production of original research. Effectiveness research can be characterized as replication research rather than original research.
4. Most research is supported by grants from federal and state agencies. The typical funding cycle for these grants is usually three years, which is often too short a period of time to complete effectiveness research.
5. Most universities take some of the money awarded in a grant as fees for administrative support of the grant. Often these fees range up toward 50% of the total grant funding. A consequence of this policy is that less money is actually available for the actual research and, generally, effectiveness research requires greater financial resources than efficacy research.

All of these factors and others contribute to the relative lack of effectiveness research. If we are ever to establish a body of knowledge base about effective interventions then it will be necessary to find mechanisms that encourage this type of research. Efficacy research is necessary but not sufficient to establish an evidence base. Effectiveness research is required to answer some of the questions that are necessary before claims of general effectiveness can be made.