School & District Management

‘What Works’ Broadens Its Research Standards

By Sarah D. Sparks — October 19, 2010 8 min read
  • Save to favorites
  • Print

For the first time since its inception, the U.S. Department of Education’s What Works Clearinghouse is broadening its definition of “gold standard” research to include nonexperimental designs.

As part of the Institute of Education Sciences’ push to make research more relevant to educators, the clearinghouse has devised standards by which it can consider two new methods for rigorous research. In addition to the randomized, controlled trials now considered the gold standard, education researchers are now able to use regression-discontinuity, a method that uses a cutoff point rather than random assignment to form comparison groups, and single-case studies in certain situations. In clearinghouse reviews, studies that adhere to the new standards are now considered, along with randomized, controlled trials, to result in the “strongest evidence” to guide educators and policymakers.

The standards, which have been in development for more than three years, were posted on the clearinghouse website over the summer and are already being used to evaluate incoming research, according to Rebecca A. Maynard, the IES commissioner for the National Center for Education, Evaluation and Regional Assistance. However, the clearinghouse plans to tweak the standards further this fall and has asked researchers to weigh in on their usability. The clearinghouse plans to produce reports of research that meets the new quality standards, but Ms. Maynard said it may be several months before the first ones come out because there will be an extra layer of review for those reports.

“We want to make sure those standards seem to have been functionally easy to apply and that we’ve been able to apply them fairly and report the evidence in a meaningful way,” she said.

The institute, which serves as the research arm for the Education Department, established the clearinghouse in 2002 in order to showcase high-quality research for educators and policymakers. Yet its rigorous criteria for inclusion, which focus primarily on randomized, controlled trials and quasi-experimental studies, at first found relatively few studies that met the bar. Of those that met the criteria, so few showed strong positive effects that the site was given the moniker the “Nothing Works Clearinghouse.”

Critics have eased up on the nickname, however, as more studies with positive effects came out and the clearinghouse moved to include a broader array of research. For example, it produces guides showing the best practices culled from all research in a given instruction area, such as English-language learners. It also conducts quick reviews of the quality of individual studies on hot topics, such as charter schools.

JAMES W. KOHLMOOS wants to see standards for many different study methods.

“I think the What Works Clearinghouse in a way helps shape the marketplace of research; the standards that are contained in the What Works Clearinghouse then shape the way people will go about doing work,” said James W. Kohlmoos, the president of the Knowledge Alliance, a Washington-based group that represents research organizations. “Just the fact that they’re looking at it, to me, suggests they are eager to make the What Works Clearinghouse more usable and more relevant to what the end user needs and wants, and, to me, that’s a positive thing.”

Different Methods

To meet the highest evidence bar previously, researchers generally had to conduct a randomized, controlled trial, in which subjects are randomly assigned to receive an intervention or not, with other factors as closely matched as possible. If the students in the two groups are otherwise the same, the program or intervention being studied can be presumed to cause changes seen between the two groups. Charter schools have become hot topics for experimental design in part because the lotteries used to assign students to overenrolled schools create a natural experiment situation for researchers.

Single-case design studies are much more specialized. They are experiments that involve a single subject, be it a student, classroom, or school district. The researcher repeatedly measures an outcome in response to an intervention under many different conditions. In this case, the researcher sets a baseline measurement before implementing the intervention, so the subject becomes its own control group.

Single-case studies are frequently used in special education research, particularly for less common disorders such as autism, according to Scott Cody, a deputy director of the clearinghouse and the associate director of human-services research at the Princeton, N.J.-based Mathematica Policy Research Inc., which operates the clearinghouse.

“Each single case study is not designed for generalization,” Mr. Cody said, but, “once you’ve got a critical mass of single-case studies, you can start to generalize.”

For the clearinghouse to publish single-case study results, they have to come from at least five studies by three or more research teams in three or more geographic locations and cover at least 20 cases, Mr. Cody said. The clearinghouse is conducting a pilot study this fall to gauge the size of effects that can be expected from those studies.

The clearinghouse also decided to write standards for regression-discontinuity studies “because it’s becoming very popular, ... so it seemed important to take a hard look,” Ms. Maynard said. “We’re getting out a little bit in front and hoping we’re able to provide guidance.”

Like randomized, controlled trials, regression-discontinuity studies gauge whether a program or intervention causes certain effects—for example, whether a phonics tutoring program increases students’ performance on a reading test. Rather than randomly assigning similar students to receive the intervention or not, a regression-discontinuity study compares students on either side of an objective cutoff point used to assign students to one group or the other. The tutoring program, say, may only take students who have scored lower than 75 percent on a phonics test. The students who scored 74 percent, barely qualifying for the tutoring, and those who scored 76 percent, just out of the pool, start from a nearly identical point, statistically. Researchers then compare the outcomes of the two groups.

“Here you know exactly who got the treatment and why, and you know that any other differences between the groups are because of the treatment,” said Jill Constantine, a clearinghouse deputy director and an associate research director at Mathematica.

Regression discontinuity can be more palatable to parents than randomized trials, since researchers don’t have to prevent students from receiving an intervention solely for research purposes. Ms. Maynard said researchers also see such studies as “easier, quicker and cheaper; people are defaulting to that instead of randomized trials.”

A Broader Base

Jon Baron, the president of the Washington-based Coalition for Evidence-Based Policy and the vice chairman of the National Board for Education Sciences, which advises the institute on clearinghouse operations, voiced concern that the clearinghouse may be jumping the gun in declaring that regression-discontinuity or single-case studies can provide results as reliable as randomized trials do. He noted that there have been few attempts to replicate the results of randomized trials using other methods, and those that have been done “show consistently [that] many of the widely used quasi-experimental methods, [including regression discontinuity], don’t do well in replicating the results of a really good randomized, control trial.”

“Those nonrandomized studies are very important, but it’s when the question comes to, is there definitive evidence that a program works, … the answer is generally you can only get to that level of evidence with well-run [randomized, controlled trials] carried out in real-world settings.”

By contrast, Miguel Urquiola, an associate economics professor at Columbia University who has previously critiqued the quality of regression-discontinuity studies, said the new standards do “a good job of simplifying some criteria that could help researchers understand when there are problems. … Under these conditions, RD can give answers that are as credible as an experiment.”

ERIC A. HANUSHEK says the standards will improve research quality.

Others, including Eric A. Hanushek, the chairman of the National Board for Education Sciences and an economist at Stanford University, see the new methods as a natural broadening of the clearinghouse’s research base. Mr. Hanushek and Mr. Urquiola both predicted the new standards will improve the quality of regression-discontinuity and single-case studies and encourage researchers to explore new questions using the methods.

“People now accept that rigorous methods can be applied to education problems, that scientific methods can be applied to education and should be,” Mr. Hanushek said.

Moreover, Ms. Maynard was quick to point out that the new methods will not replace experimental designs in IES’s research quiver, as techniques for experimental designs continue to evolve. “Most of the arguments against using [randomized, controlled trials] are a function of not thinking broadly about how you design an experiment,” she said.

Mr. Urquiola agreed. “It opens the field to certain types of questions that [regression-discontinuity] experiments aren’t set to deal with. In education, there are so many questions that one approach can lead to things being quite limited.

“Wayne J. Camara, the president of the National Council on Measurement in Education, said researchers have been waiting for the clearinghouse to broaden its scope in this way for many years. He considers the new standards a good first step, but both he and Mr. Kohlmoos said they would like to see standards to signify high-quality research conducted in many different methods, from experimental to descriptive.

“Only when you see repeated evidence that looks at context and uses different methodologies and perspectives can you … really understand whether interventions will really hold up against the test of time,” Mr. Camara said.

A version of this article appeared in the October 20, 2010 edition of Education Week as Federal Criteria for Studies Grow


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
Strategies for Incorporating SEL into Curriculum
Empower students to thrive. Learn how to integrate powerful social-emotional learning (SEL) strategies into the classroom.
Content provided by Be GLAD
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Leadership in Education: Building Collaborative Teams and Driving Innovation
Learn strategies to build strong teams, foster innovation, & drive student success.
Content provided by Follett Learning
School & District Management K-12 Essentials Forum Principals, Lead Stronger in the New School Year
Join this free virtual event for a deep dive on the skills and motivation you need to put your best foot forward in the new year.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management What the Research Says 5 Things Schools Can Do This Summer to Improve Student Attendance Next Year
Schools can get a jump on student attendance during the school year by using data, leveraging summer programs, and connecting with families.
6 min read
Julian Gresham, 12, left, works in a group to program a Bee-Bot while in their fifth grade summer school class Monday, June 14, 2021, at Goliad Elementary School. Bee-bots and are new to Ector County Independent School District and help to teach students basic programming skills like sequencing, estimation and problem-solving.
Julian Gresham, 12, left, works on a robotics programming activity in a 5th-grade summer school class June 14, 2021, at Goliad Elementary School in Ector County, Texas. Active summer programs may improve students' attendance during the school year.
Jacob Ford/Odessa American via AP
School & District Management Grad Rates Soared at a School Few Wanted to Attend. How It Happened
Leaders at this Florida high school have "learned to be flexible" to improve graduation rates.
8 min read
Student hanging on a tearing graduate cap tassel
DigitalVision Vectors/Getty
School & District Management Opinion Don’t Just Listen to the Loudest Voices: Resources for Ed. Leaders
These resources can help school and district leaders communicate with their communities.
Jennifer Perry Cheatham & Jenny Portillo-Nacu
5 min read
A pair of hands type on a blank slate of keys that are either falling apart or coming together on a bed of sharpened pencils.  Leadership resources.
Raul Arias for Education Week
School & District Management The Harm of School Closures Can Last a Lifetime, New Research Shows
The short-term effects on students when their schools close have been well documented. New research examines the long-term impact.
5 min read
Desks and chairs are stacked in an empty classroom after the permanent closure of Queen of the Rosary Catholic Academy in Brooklyn borough of New York on Aug. 6, 2020.
Desks and chairs are stacked in an empty classroom after the permanent closure of Queen of the Rosary Catholic Academy in Brooklyn borough of New York on Aug. 6, 2020. A new study examines the long-term effects on students whose schools close.
Jessie Wardarski/AP