What Works Offers ‘Quick Reviews’ of New Studies

By Debra Viadero — June 10, 2008 4 min read

In an effort to make its services more timely and relevant, the federally supported What Works Clearinghouse has launched a series of “quick reviews” to assess the methodological soundness of studies that have made national news.

“There are all kinds of studies coming out daily that are cutting-edge,” said Mark Dynarski, who directs the clearinghouse, which is based at Mathematica Policy Research Inc. in Princeton, N.J. “The idea is whether the clearinghouse can help the public in determining whether studies are well crafted, if they use sound methods and make good inferences.”

Created in 2002 by the U.S. Department of Education’s Institute of Education Sciences, the online clearinghouse vets the research base for programs, practices, or policies that educators might try out in their own jurisdictions. But the first reviews were slow in coming, and few studies met the clearinghouse’s strict evidence standards, leading some critics to dub the project the “nothing works” clearinghouse.

The clearinghouse plans to post the new quick reviews, in contrast, within a month of a study’s public release. The turnaround time is faster because reviewers are evaluating a single study, rather than a body of research.

The first five reviews, posted online starting late last month, give mixed assessments of recent studies on after-school programs, a student-athlete drug testing program, a cash-incentive program for teachers and students of Advanced Placement courses, a computer-based program for teaching algebra, and on whether 6th graders are better off in middle schools or elementary schools.

Some of the reports are already drawing sharp criticism from study authors.

Research Into Practice: The Push For Usability
This four-part series examines the movement to make education research more “usable” and explores some efforts to connect the worlds of research and practice.

“This is all politics. I’m telling you from the bottom of my heart,” said Dr. Linn Goldberg, a professor of medicine at Oregon Health and Science University, in Portland. Clearinghouse reviewers determined that the study Dr. Goldberg led on school-based drug testing for student-athletes was “not consistent with WWC evidence standards” —the lowest of three possible ratings.

The other rating choices are “consistent with WWC evidence standards” and “consistent with WWC evidence standards with reservations.”

Tough Standards

In the study, published in November in the peer-reviewed Journal of Adolescent Health, Dr. Goldberg and his colleagues evaluated the effects of a drug-testing program in which high school student-athletes were randomly selected for unannounced tests for alcohol and drug use.

After two years, the researchers found, students taking part in the testing were no less likely than their control-group counterparts to be using prohibited substances. Dr. Goldberg said the findings, which were widely reported in the press in February, run counter to the Bush administration’s stance favoring such programs.

In their review, What Works Clearinghouse analysts said the findings were inconclusive, in part, because of the high attrition rates among the schools and students involved. Seven of the 18 schools that signed up for the study dropped out, according to the clearinghouse’s report. In addition, 29 percent of students in the drug-testing program and 19 percent of the control-group students failed to return final surveys.

However, Dr. Goldberg said that the schools withdrew before the drug-testing programs began in any school, and that they did so in response to a threatened lawsuit. Final student data are missing, Dr. Goldberg added, because students either graduated or quit their teams the next year.

“It’s natural attrition,” he said, “and it’s much lower than we expected.”

For his part, Mr. Dynarski of the What Works Clearinghouse denied that the study was selected to serve as a political target. He said reviewers chose it because it was published within the time frame the clearinghouse had set for its first round of reviews. Although federal education officials can refer studies to the clearinghouse under the new “quick review” guidelines, none of the studies in the first round came to the clearinghouse by that route, he added.

“Attrition is a legitimate concern,” Mr. Dynarski said. “Clearinghouse standards pay a lot of attention to attrition, and here there is a 10-percentage-point difference between the control and experimental groups.”

The clearinghouse also gives a “not consistent” rating to an eight-state study of 35 high-quality after-school programs that was first reported in Education Week last November. (“High-Quality After-School Programs Tied to Test-Score Gains,” Nov. 28, 2007.) Deborah Lowe Vandell, the study’s lead author and the chairwoman of the education department at the University of California, Irvine, said the reviewers’ determination was not surprising, given the narrow range of study designs that pass the clearinghouse’s evidence screens.

“A clinical trial [one of the designs that pass muster] is an important standard, but it represents only one type of research evidence,” she said.

Clearinghouse reviewers gave a “consistent” rating to a randomized study that found positive academic effects for students using SimCalc Mathworlds, a software program for teaching algebra concepts. Two other studies—one on a Texas effort to give cash bonuses to students and teachers for high scores on AP tests and the other on the effects of teaching 6th graders in middle schools—were rated “consistent with WWC evidence standards with reservations.”

A version of this article appeared in the June 11, 2008 edition of Education Week as What Works Offers ‘Quick Reviews’ of New Studies


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Educator-Driven EdTech Design: Help Shape the Future of Classroom Technology
Join us for a collaborative workshop where you will get a live demo of GoGuardian Teacher, including seamless new integrations with Google Classroom, and participate in an interactive design exercise building a feature based on
Content provided by GoGuardian
School & District Management Live Online Discussion A Seat at the Table With Education Week: What Did We Learn About Schooling Models This Year?
After a year of living with the pandemic, what schooling models might we turn to as we look ahead to improve the student learning experience? Could year-round schooling be one of them? What about online
School & District Management Webinar What's Ahead for Hybrid Learning: Putting Best Practices in Motion
It’s safe to say hybrid learning—a mix of in-person and remote instruction that evolved quickly during the pandemic—is probably here to stay in K-12 education to some extent. That is the case even though increasing

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Miguel Cardona: Schools Must Work to Win Trust of Families of Color as They Reopen
As Secretary of Education Miguel Cardona announced new school reopening resources, he encouraged a focus on equity and student engagement.
4 min read
Education Secretary nominee Miguel Cardona testifies before the Senate Health, Education, Labor and Pensions committee during his confirmation hearing Feb. 3, 2021.
Now-U.S. Secretary of Education Miguel Cardona testifies before the Senate Health, Education, Labor and Pensions committee during his confirmation hearing in February.
Susan Walsh/AP
Federal CDC: Nearly 80 Percent of K-12, Child-Care Workers Have Had at Least One COVID-19 Shot
About four out of five teachers, school staffers, and child-care workers had first COVID-19 vaccine doses by the end of March, CDC says.
2 min read
John Battle High School teacher Jennifer Daniel receives her COVID-19 vaccine on Jan. 11, 2021. Teachers received their first vaccine during an all-day event at the Virginia Highlands Higher Education Center in Abingdon, Va.
John Battle High School teacher Jennifer Daniel receives her COVID-19 vaccine on Jan. 11at the Virginia Highlands Higher Education Center in Abingdon, Va.
David Crigger/Bristol Herald Courier via AP
Federal Ed. Dept. to Review Title IX Rules on Sexual Assault, Gender Equity, LGBTQ Rights
The review could reopen a Trump-era debate on sexual assault in schools, and it could spark legal discord over transgender student rights.
4 min read
Symbols of gender.
Federal Q&A EdWeek Q&A: Miguel Cardona Talks Summer Learning, Mental Health, and State Tests
In an interview after a school reopening summit, the education secretary also addressed teachers' union concerns about CDC guidance.
10 min read
Education Secretary Miguel Cardona speaks during a press briefing at the White House on March 17, 2021.
U.S. Secretary of Education Miguel Cardona speaks during a press briefing at the White House on March 17.
Andrew Harnik/AP