research is conducted to answer questions about the impact of an
intervention as it is “taken to scale” and implemented in typical
When does it work?
Effectiveness research is primarily concerned with answering questions
about the impact of an intervention as it is implemented in typical
practice settings. As an intervention is moved from research settings
to practice settings there are several variables that can influence the
overall impact of an intervention.
Typically, in effectiveness
research the unit of analysis is populations of students, classrooms,
or schools. As a result the impact of an intervention on an individual
student is obscured. One of the primary concerns of effectiveness
research is how broad an impact will an intervention have so the impact
on an individual student is less important.
effectiveness research is concerned with the impact on a population or
setting, there is a generalization from the research setting to a
specific practice setting that is required. To the extent that the
practice setting is similar to the research setting then we can have
greater confidence in the impact of an intervention. Nonetheless it is
a generalization and as such, it can be less than perfect.
research answers questions about the probability of impact with a
particular population or in a specific setting if the intervention is
implemented as prescribed but it does not answer questions about how to
actually implement the intervention.
Effectiveness research is
less common than efficacy research. The reason for this is that there
are some institutional and logistical barriers that make effectiveness
research more difficult.
an intervention has been demonstrated to be promising, the function of
effectiveness research is to determine how robust the intervention is
when it is implemented in settings similar to typical practice
settings. Among the dimensions that have to be considered as an
intervention is taken to scale are:
- Which populations are likely to respond to the intervention?
- Which populations will not benefit?
- What are the characteristics of the setting are necessary for an intervention to be effective?
- What level of training is required for those implementing the intervention?
- What level of treatment integrity is necessary to produce beneficial effects?
- What “dose” of the intervention is necessary?
- How acceptable is the intervention to those who are responsible for implementation?
- How powerful are the effects of this intervention compared to other interventions?
- What are the typical costs of implementing the intervention?
|Effectiveness Research and the Unit of Analysis
||Back to Top
research is still research so the standards of scientific rigor are
still required. Often the unit of analysis changes as we move from
efficacy to effectiveness research. In efficacy research, the unit of
analysis is often the performance on individual students. In
effectiveness research the unit of analysis is at a larger scale such
as populations of students, classrooms, schools, characteristics of
staff, etc. The impact on an individual student is less clear in this
level of analysis but the impact on the larger units is better
understood. This change in the unit of analysis highlights the
fundamental difference between efficacy and effectiveness research.
Each type of research is designed to answer different questions.
research does not answer questions about how effective an intervention
will be in a specific setting but it can guide decision makers to
consider the characteristics of the population and the settings in
which the research was conducted relative to the characteristics of the
populations and settings in which the decision maker is concerned. The
more similar the research setting and the practice setting, the greater
confidence we can have that the intervention will produce positive
effects; however, it is still a generalization from the research to
practice settings so there will be occasions in which an effective
intervention will not result in positive outcomes in a particular
practice setting. Generalizations from one context to another are often
less than perfect.
Effectiveness research does not answer how to
accomplish implementation within a specific setting. Effectiveness
research only identifies interventions that are likely to produce
positive outcomes. Questions of how to implement an intervention are
answered by other lines of research. The transition from research to
practice is one of the biggest challenges facing those interested in
developing a culture of evidence-based decision-making within education.
Relative to efficacy research there is little effectiveness research being conducted. There are several reasons for this:
- Effectiveness research generally requires greater numbers of participants than efficacy research.
The greater numbers are required to establish that an intervention is
generally effective with a population. Recruiting greater number of
participants and actually exposing them to the experimental
intervention generally requires more time and effort than is required
of efficacy research.
- University professors operating under the rules of promotion and tenure conduct most research.
Generally, these rules require a high level of productivity as measured
by publication in professional journals and effectiveness research
often takes more time per study than efficacy research.
- The rules for promotion and tenure encourage production of original research. Effectiveness research can be characterized as replication research rather than original research.
- Most research is supported by grants from federal and state agencies.
The typical funding cycle for these grants is usually three years,
which is often too short a period of time to complete effectiveness
- Most universities take some of the money awarded in a grant as fees for administrative support of the grant.
Often these fees range up toward 50% of the total grant funding. A
consequence of this policy is that less money is actually available for
the actual research and, generally, effectiveness research requires
greater financial resources than efficacy research.
of these factors and others contribute to the relative lack of
effectiveness research. If we are ever to establish a body of knowledge
base about effective interventions then it will be necessary to find
mechanisms that encourage this type of research. Efficacy research is
necessary but not sufficient to establish an evidence base.
Effectiveness research is required to answer some of the questions that
are necessary before claims of general effectiveness can be made.