Federal

‘What Works’ Process for Assessing Studies Called Valid

Officials buoyed, critics unsatisfied, by review requested by Congress.
By Debra Viadero — November 21, 2008 4 min read
  • Save to favorites
  • Print

Includes updates and/or revisions.

A new review by an independent panel of experts concludes that the U.S. Department of Education’s much-criticized What Works Clearinghouse is doing a “reasonable job” of reviewing and rating the research evidence on the effectiveness of programs and practices in education.

Created in 2002 by the Institute of Education Sciences, the department’s primary research arm, the clearinghouse has come under fire from policymakers, researchers, and practitioners, who question its usefulness and methods. Some have dubbed it the “nothing works” clearinghouse because of the limited number of programs and studies that meet its strict screening standards.

But in its report, which was posted online Nov. 19, the six-member panel contends that the clearinghouse’s review standards are “well documented” and “reasonable.” The study further characterizes the reports that the clearinghouse produces as “succinct and meaningful.”

Noting that their review focused narrowly on whether the clearinghouse makes judgments that are scientifically valid, the panelists also call, however, for a fuller look at the entire mission of the enterprise.

Both Grover J.“Russ” Whitehurst, the outgoing director of the IES, and Robert C. Granger, who chairs the national board that advises the research agency, called the panel’s findings reassuring.

“In a marketplace that is unsophisticated with regard to research quality, there has to be an entity that uses rigorous standards to vet research on education program effectiveness for practitioners and policymakers,” Mr. Whitehurst wrote in a letter accompanying the report.

Mixed Reactions

Members of the research community offered mixed reactions.

“For me, the key question is how usable is the information that the clearinghouse produces,” added James W. Kohlmoos, the president of the Knowledge Alliance, a Washington group that represents research organizations. “We haven’t answered that yet.”

Robert E. Slavin, a researcher and co-founder of the Baltimore-based Success for All Foundation, concurred. “The panel acknowledged that it was given too little time and too narrow a mandate to adequately evaluate the WWC,” Mr. Slavin, who has been an outspoken critic of the clearinghouse, said in an e-mail.

Begun in late July, the review was commissioned by the National Board for Education Sciences, the presidentially appointed panel that advises the IES.

Mr. Granger, the board’s chairman, said the House Appropriations panel that oversees education programs called for a more comprehensive investigation by the U.S. Government Accountability Office, the watchdog agency for Congress. On the Senate side, though, appropriators requested a more focused look at the scientific validity of the clearinghouse’s review procedures.

“There was just not enough time with the current board to take on a broader review,” Mr. Granger said, noting that the terms of five members of the board, him included, are due to end later this month. While replacement members have been named, their nominations are not likely to be approved by the Senate before the new administration takes control of the White House in January. Currently, only 11 of the 15 slots on the board are filled.

“The focus is narrow by design,” added Mr. Granger, who is the president of the William T. Grant Foundation in New York City. “It is the crux of the matter. If you can’t trust the information that’s on the What Works site, then you can’t trust anything else it does.”

Changes Recommended

Mr. Granger, working with staff members at the IES, helped select the review-panel members, most of whom are experts at synthesizing research findings in fields other than education.

The panelists are: C. Hendricks Brown, a biostatistician at the University of South Florida in Tampa; David Card, an economist at the University of California, Berkeley; Kay Dickerson, an epidemiologist at Johns Hopkins University in Baltimore; Joel B. Greenhouse, a biostatistician at Carnegie Mellon University in Pittsburgh; Jeffrey R. Kling, an economist at the Brookings Institution, a think tank in Washington; and Julia H. Littell, a professor of social work and social research at Bryn Mawr College in Bryn Mawr, Pa.

Besides calling for a more comprehensive study of the clearinghouse, the panel also urged the research agency to set up a process for regular reviews of the clearinghouse standards—an action that Mr. Whitehurst said the IES would undertake.

Among its other recommendations, the panel called on the clearinghouse to: review its standards regarding the attrition rates of subjects who take part in experiments; establish a formal process for tracking potential conflicts of interest in the studies it reviews, especially when program developers pay for studies of their own programs; and take another look at the standards it uses to account for cases when subjects fail to comply with the intervention being studied or when intervention practices cross over from the experimental to the control group.

A version of this article appeared in the December 03, 2008 edition of Education Week

Events

Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!


Content provided by Panorama
Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal 'A Snitch Line on Parents.' GOP Reps Grill AG Over Response to Threats on School Officials
Attorney General Merrick Garland said his effort is meant to address violent threats against school boards, not to stifle parents' dissent.
5 min read
LEFT: Attorney General Merrick Garland speaks during a House Judiciary Committee oversight hearing of the Department of Justice on Capitol Hill in Washington, Thursday, Oct. 21, 2021. RIGHT: Rep. Jim Jordan, R-Ohio, questions Attorney General Merrick Garland.
Attorney General Merrick Garland, left, speaks during a House Judiciary Committee oversight hearing of the U.S. Department of Justice on Capitol Hill on Thursday, questioned by Rep. Jim Jordan, R-Ohio, right, among others.
Greg Nash via AP, Andrew Harnik/AP
Federal School Boards, 'Domestic Terrorism,' and Free Speech: Inside the Debate
From critical race theory to COVID policy, the heat on schools has raised issues involving free speech and the safety of public officials.
13 min read
Brenda Stephens, a school board member with Orange County Public Schools in Hillsborough, N.C. has purchased a weapon and taken a concealed carry class over concerns for her personal safety.
Brenda Stephens, a school board member in Hillsborough, N.C., says board members face threats and bullying, an atmosphere far different from what she's encountered in years of board service.
Kate Medley for Education Week
Federal Senate Confirms Catherine Lhamon to Civil Rights Post; Kamala Harris Casts Decisive Vote
Joe Biden's controversial pick to lead the Education Department's office for civil rights held that job in the Obama administration.
2 min read
Catherine Lhamon, nominee to be assistant secretary for civil rights at the Department of Education, testifies during a Senate Health, Education Labor and Pensions Committee confirmation hearing in Dirksen Building on Tuesday, July 13, 2021.
Catherine Lhamon, then-nominee to be assistant secretary for civil rights at the U.S. Department of Education, testifies during a Senate Health, Education Labor and Pensions Committee confirmation hearing in July.
Tom Williams/CQ Roll Call via AP Images
Federal White House Outlines COVID-19 Vaccination Plans for Kids 5-11
The Biden administration will rely on schools, pharmacies, and pediatricians to help deliver the COVID-19 shots to younger children.
3 min read
Ticket number 937 sits on a COVID-19 vaccination at the drive-thru vaccination site in the Coweta County Fairgrounds on Jan. 14, 2021, in Newnan, Ga.
A ticket number sits on a COVID-19 vaccination at the drive-thru vaccination site in the Coweta County Fairgrounds in Newnan, Ga.
Curtis Compton/Atlanta Journal-Constitution via AP