School & District Management

Researchers Question Clearinghouse Choices

By Debra Viadero — August 11, 2004 4 min read

The What Works Clearinghouse, the Department of Education’s newly functioning enterprise to give Consumer Reports-style ratings on research and educational programs, is getting mixed reviews so far from one key group: the researchers whose work it features.

While a few among that small group of scholars are happy to see their work reaching a wider audience, some also express concerns about the way it is being presented. They say reviewers are misinterpreting and pigeonholing their studies and sometimes inadvertently casting aspersions on potentially useful research.

“When I looked at it, I was just kind of appalled,” said G. Michael Pressley, the director of doctoral programs in teacher education at Michigan State University in East Lansing. A study he co-wrote on a practice known as reciprocal teaching is among the handful of studies that got a thumbs-up from clearinghouse reviewers when the site was unveiled last month. (“‘What Works’ Research Site Unveiled,” July 14, 2004.)

For their part, clearinghouse researchers say they are already working to address some of the scholars’ concerns. They are drafting text for the Web site, for instance, to explain better why some studies are listed as failing to pass muster, and are working as quickly as possible to add more studies to their online archives.

The reviewers plan for now, though, to continue using the strict methodological criterion that is drawing much of the criticism for weeding out useful research. Emulating standards set by medical research, the clearinghouse puts a premium on randomized field trials, in which subjects are randomly to either control or experimental groups. As a result, of 18,000 studies reviewed, only 12 so far have fully met clearinghouse standards.

“Random assignment is a strong design,” said Rebecca S. Herman, the project director. “It’s always been a strong design, and I think it continues to be.”

‘Does Not Pass Screen’

Some of the harshest criticism has come from Mr. Pressley, who in letters to the clearinghouse and to Education Week has charged that the reviewers mistakenly categorized his study on reciprocal teaching as a trial of peer tutoring. He says peer tutoring is just one part of that approach.

“There is no way to draw a conclusion about peer tutoring or cooperative learning per se from this study,” he writes.

In a written reply to Mr. Pressley, Ms. Herman and Robert F. Boruch, the project’s principal investigator, said they included the study because they wanted to offer educators as much information as possible on peer-assisted learning.

But Mr. Pressley, like other researchers, also raised questions about the 173 peer-tutoring studies that did not make the grade. They are listed on the Web site under the heading “does not pass screen,” which means that they were either deemed irrelevant or did not meet the clearinghouse’s methodological standards. The problem, the researchers said, was that the site does not explain why those studies were rejected.

“Many studies that ‘do not pass the screen’ may be viewed as not valuable when in fact they may be extremely helpful in understanding or adapting an intervention to a new context,” said Anthony J. Gabriele, a researcher at the University of Northern Iowa in Cedar Falls whose own study landed in that category. The label did not surprise him, he said, because the study was never designed to answer whether peer tutoring works. But he became concerned when a friend sent an e-mail expressing condolences on the study’s categorization.

“To the extent that this classification discourages stakeholders from looking at these studies, I think we may be providing too narrow a focus given our relative understanding regarding what works,” Mr. Gabriele said.

James J. Baker, the developer of a middle school mathematics program known as Expert Mathematician, is also dismayed at the way his research on the program is reported. His study—the only one that fully met the criteria for his topic—used a random-assignment strategy to test whether students could learn as much with his student-driven, computer-based math program as they could from a traditional, teacher-directed curriculum known as Transition Mathematics. The problem, he argues, is that the Web site says his program had no effect without explaining that students made learning gains in both groups.

Ms. Herman said the clearinghouse could not provide that context because it had no research to show that Transition Mathematics works better than other curricula. “Without that, we couldn’t report out that it was effective,” she said. Ms. Herman said other analyses in the study—showing, for instance, that students’ attitudes toward math improved more with Expert Mathematician—did not meet clearinghouse criteria.

In all, she said, the clearinghouse has received about 100 comments on the new site, which took two years to develop. Some suggested that the screening criteria had not gone far enough in weeding out the “best of the best” research in the field. Ms. Herman said some feedback came from the practitioners the clearinghouse was designed to serve.

“They said that we were providing the kind of critical look that helps them figure out whether a study is useful or not,” she added.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Recruitment & Retention Webinar
Recruiting and Retaining a More Diverse Teaching Workforce
We discuss the importance of workforce diversity and learn strategies to recruit and retain teachers from diverse backgrounds.
Content provided by EdWeek Top School Jobs
Student Well-Being Webinar Boosting Teacher and Student Motivation During the Pandemic: What It Takes
Join Alyson Klein and her expert guests for practical tips and discussion on how to keep students and teachers motivated as the pandemic drags on.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Holistic Approach to Social-Emotional Learning
Register to learn about the components and benefits of holistically implemented SEL.
Content provided by Committee for Children

EdWeek Top School Jobs

Principal
Meredith, New Hampshire
Inter-Lakes School District
Elementary Principal
Washington State
Wenatchee School District
Principal
Meredith, New Hampshire
Inter-Lakes School District
Elementary Principal
Washington State
Wenatchee School District

Read Next

School & District Management Superintendent Who Led During COVID-19 School Shutdowns Gets Top Honors
Michelle Reid of Washington state's Northshore district, one of the very first to close schools last March, was named National Superintendent of the Year.
3 min read
Michelle Reid, superintendent of the Northshore district in Washington
Michelle Reid, the superintendent of the Northshore district in Washington, was named National Superintendent of the Year.
courtesy of AASA, the School Superintendents Association
School & District Management Is Lunchtime the 'Weak Link' in School Reopening Plans?
It's risky when students are inside and unmasked, experts say. Here are five ways to mitigate that risk and make in-school meals safer.
11 min read
Elementary students in Brownsville, Texas, eat a socially distanced lunch in the school cafeteria. Experts say there are ways to mitigate the risk of contracting COVID-19 even when kids take their masks off to eat.
Elementary students in Brownsville, Texas, eat a socially distanced lunch in the school cafeteria. Experts say there are ways to mitigate the risk of contracting COVID-19 when kids take their masks off to eat.
Denise Cathey/The Brownsville Herald via AP
School & District Management Los Angeles Unified Shrinks Police Budget. Money Will Support Black Student Achievement
The board overseeing the Los Angeles Unified School District has cut $25 million from the budget for school police and will use the money to help fund an achievement plan for Black students.
1 min read
Demonstrators holds signs during a protest to demand the defunding of the Los Angeles school district police outside of the school board headquarters on June 23, 2020, in Los Angeles.
Demonstrators holds signs during a protest to demand the defunding of the Los Angeles school district police outside of the school board headquarters on June 23, 2020, in Los Angeles.
Marcio Jose Sanchez/AP
School & District Management 6 Big Questions Superintendents Are Asking About the CDC Guidance
School leaders' queries show the challenges and concerns they face over resuming in-person teaching.
7 min read
President Joe Biden speaks during a visit to National Institutes of Health in Bethesda, Md. on Feb. 11, 2021. The Centers for Disease Control and Prevention on Friday, Feb. 12, released long-awaited guidance telling schools what measures are needed to teach in-person during the pandemic. Biden requested the updated guidance in response to complaints that the CDC’s school guidelines under the Trump administration were unclear and inconsistent.
A teacher works with a student behind plexi-glass in Chicago. Superintendents asked CDC officials this week whether using plexi-glass would allow them to shorten social distances in classrooms.
Ashlee Rezin Garcia/Chicago Sun-Times via AP