What Works Offers ‘Quick Reviews’ of New Studies

By Debra Viadero — June 10, 2008 4 min read
  • Save to favorites
  • Print

In an effort to make its services more timely and relevant, the federally supported What Works Clearinghouse has launched a series of “quick reviews” to assess the methodological soundness of studies that have made national news.

“There are all kinds of studies coming out daily that are cutting-edge,” said Mark Dynarski, who directs the clearinghouse, which is based at Mathematica Policy Research Inc. in Princeton, N.J. “The idea is whether the clearinghouse can help the public in determining whether studies are well crafted, if they use sound methods and make good inferences.”

Created in 2002 by the U.S. Department of Education’s Institute of Education Sciences, the online clearinghouse vets the research base for programs, practices, or policies that educators might try out in their own jurisdictions. But the first reviews were slow in coming, and few studies met the clearinghouse’s strict evidence standards, leading some critics to dub the project the “nothing works” clearinghouse.

The clearinghouse plans to post the new quick reviews, in contrast, within a month of a study’s public release. The turnaround time is faster because reviewers are evaluating a single study, rather than a body of research.

The first five reviews, posted online starting late last month, give mixed assessments of recent studies on after-school programs, a student-athlete drug testing program, a cash-incentive program for teachers and students of Advanced Placement courses, a computer-based program for teaching algebra, and on whether 6th graders are better off in middle schools or elementary schools.

Some of the reports are already drawing sharp criticism from study authors.

Research Into Practice: The Push For Usability
This four-part series examines the movement to make education research more “usable” and explores some efforts to connect the worlds of research and practice.

“This is all politics. I’m telling you from the bottom of my heart,” said Dr. Linn Goldberg, a professor of medicine at Oregon Health and Science University, in Portland. Clearinghouse reviewers determined that the study Dr. Goldberg led on school-based drug testing for student-athletes was “not consistent with WWC evidence standards” —the lowest of three possible ratings.

The other rating choices are “consistent with WWC evidence standards” and “consistent with WWC evidence standards with reservations.”

Tough Standards

In the study, published in November in the peer-reviewed Journal of Adolescent Health, Dr. Goldberg and his colleagues evaluated the effects of a drug-testing program in which high school student-athletes were randomly selected for unannounced tests for alcohol and drug use.

After two years, the researchers found, students taking part in the testing were no less likely than their control-group counterparts to be using prohibited substances. Dr. Goldberg said the findings, which were widely reported in the press in February, run counter to the Bush administration’s stance favoring such programs.

In their review, What Works Clearinghouse analysts said the findings were inconclusive, in part, because of the high attrition rates among the schools and students involved. Seven of the 18 schools that signed up for the study dropped out, according to the clearinghouse’s report. In addition, 29 percent of students in the drug-testing program and 19 percent of the control-group students failed to return final surveys.

However, Dr. Goldberg said that the schools withdrew before the drug-testing programs began in any school, and that they did so in response to a threatened lawsuit. Final student data are missing, Dr. Goldberg added, because students either graduated or quit their teams the next year.

“It’s natural attrition,” he said, “and it’s much lower than we expected.”

For his part, Mr. Dynarski of the What Works Clearinghouse denied that the study was selected to serve as a political target. He said reviewers chose it because it was published within the time frame the clearinghouse had set for its first round of reviews. Although federal education officials can refer studies to the clearinghouse under the new “quick review” guidelines, none of the studies in the first round came to the clearinghouse by that route, he added.

“Attrition is a legitimate concern,” Mr. Dynarski said. “Clearinghouse standards pay a lot of attention to attrition, and here there is a 10-percentage-point difference between the control and experimental groups.”

The clearinghouse also gives a “not consistent” rating to an eight-state study of 35 high-quality after-school programs that was first reported in Education Week last November. (“High-Quality After-School Programs Tied to Test-Score Gains,” Nov. 28, 2007.) Deborah Lowe Vandell, the study’s lead author and the chairwoman of the education department at the University of California, Irvine, said the reviewers’ determination was not surprising, given the narrow range of study designs that pass the clearinghouse’s evidence screens.

“A clinical trial [one of the designs that pass muster] is an important standard, but it represents only one type of research evidence,” she said.

Clearinghouse reviewers gave a “consistent” rating to a randomized study that found positive academic effects for students using SimCalc Mathworlds, a software program for teaching algebra concepts. Two other studies—one on a Texas effort to give cash bonuses to students and teachers for high scores on AP tests and the other on the effects of teaching 6th graders in middle schools—were rated “consistent with WWC evidence standards with reservations.”

A version of this article appeared in the June 11, 2008 edition of Education Week as What Works Offers ‘Quick Reviews’ of New Studies


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Seamless Integrations for Engagement in the Classroom
Learn how to seamlessly integrate new technologies into your classroom to support student engagement. 
Content provided by GoGuardian
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Recruitment & Retention Webinar
Be the Change: Strategies to Make Year-Round Hiring Happen
Learn how to leverage actionable insights to diversify your recruiting efforts and successfully deploy a year-round recruiting plan.
Content provided by Frontline
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Critical Ways Leaders Can Build a Culture of Belonging and Achievement
Explore innovative practices for using technology to build an environment of belonging and achievement for all staff and students.
Content provided by DreamBox Learning

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Conservatives Hammer on Hot-Button K-12 Education Issues at Federalist Society Event
The influential legal group discussed critical race theory, gender identity, and Title IX.
6 min read
Secretary of Education Betsy DeVos speaks at the Phoenix International Academy in Phoenix on Oct. 15, 2020.
Former U.S. Secretary of Education Betsy DeVos was among a phalanx of conservatives addressing K-12 issues at a conference of the Federalist Society.
Matt York/AP
Federal Cardona Back-to-School Tour to Focus on Teacher Pipeline, Academic Recovery
U.S. Secretary of Education Miguel Cardona will spend a week traveling to six states to highlight a range of K-12 priorities.
2 min read
Secretary of Education Miguel Cardona answers questions during an interview in his office in Washington, D.C., on Tuesday, August 23, 2022.
U.S. Secretary of Education Miguel Cardona continues a tradition of on-site visits by the nation's top education official as the school year opens.
Alyssa Schukar for Education Week
Federal Biden's Student Loan Forgiveness: How Much Will It Help Teachers?
Advocates say Black educators—who tend to carry heavier debt loads—won't benefit as much.
5 min read
Illustration of student loans.
Federal Q&A U.S. Education Secretary Cardona: How to Fix Teacher Shortages, Create Safe Schools
In an exclusive interview with Education Week, the secretary looks ahead to the challenges of this school year.
10 min read
Secretary of Education Miguel Cardona answers questions during an interview in his office in Washington, D.C., on Tuesday, August 23, 2022.
Secretary of Education Miguel Cardona answers questions during an interview in his office in Washington on Aug. 23.
Alyssa Schukar for Education Week