Standards

Release of Unreviewed Studies Sparks Debate

By Debra Viadero — May 17, 2005 6 min read
  • Save to favorites
  • Print

Researchers from two of the nation’s most eminent universities, Harvard and Stanford, recently unveiled studies on different, but equally controversial, education issues. One was on school accountability policy, the other on teacher certification, but critics’ responses were the same.

They complained that the studies were “not quite ready for prime time” because they had not undergone the independent scrutiny that usually comes when studies are published in academic journals.

It’s a refrain that has become common lately as research on hot-button issues moves directly from the author’s computer printer to a press release. The criticism has some scholars wondering whether education researchers need to do a better job of policing themselves.

“I think there have been abuses on all sides,” said James. G. Cibulka, the dean of the college of education at the University of Kentucky, in Lexington. “As a profession, we really need to have some agreed-upon criteria for vetting studies before they make their way to the public.”

Whether the buzz about the need for giving studies some level of professional review before they hit the public arena is new, or even limited to education research, is an open question.

Ready for the Public?

Studies generally undergo an evaluation process before they are accepted for publication in an academic journal, presented at a conference, or released by a think tank.

Peer review for articles published in a professional journal typically involves three or more reviewers who critique an author’s work in writing. The reviewers are “blind,” meaning that they are chosen by the journal editor and that the author does not know their identities.

Peer review for presentations at a professional conference is a less-rigorous procedure. Reviewers often accept or reject a proposed presentation on the basis of an abstract or a three- or four-page paper.

Peer review at a nonpartisan think tank, such as the New York City-based MDRC or the Washington-based American Institutes for Research, takes place mostly in-house as think tank researchers review one another’s work. In the later stages of its three-tiered review process, MDRC also enlists board members, outside experts in the topic under study, and participants from the study sites. For the Manhattan Institute, a think tank widely seen to support conservative perspectives, researcher Jay P. Greene said he typically circulates his papers to several outside colleagues before releasing them to the public.

“One of the traditions of the education research enterprise has been the high frequency of advocacy research,” said Grover J. “Russ” Whitehurst, the director of the U.S. Department of Education’s Institute of Education Sciences. “I would be skeptical that there’s been an uptick on this.”

Others disagree. They say the debate over the need for tighter peer-review mechanisms sharpened last summer after the American Federation of Teachers produced an unreviewed study suggesting that some students in public elementary schools outperformed their counterparts in the same grades in charter schools on national tests of reading and mathematics. (“AFT Charter School Study Sparks Heated National Debate,” Sept. 1, 2004.)

When the AFT study’s findings became the subject of the lead story in The New York Times on Aug. 17, supporters of charter schools mounted a counterattack that included editorials, letters to the editor, and a full-page advertisement in national newspapers. The ad, signed by 31 academics, mostly criticized the study on methodological grounds, but it also called for closer scrutiny of research before findings percolate into the mainstream media. It said that “such studies need to be vetted by independent scholars, as is commonly done in coverage of research on the biological and physical sciences.”

Since then, what has irked some other researchers is that three of the signers went on to do just what the teachers’ union had done: They took research findings to the press before the studies had undergone a formal peer-review process. That drew charges of hypocrisy from some of their research colleagues.

Jay P. Greene, one of the signers who was criticized, said he saw the ad as more of an admonition to journalists than to researchers.

“The ad never said all studies have to be peer-reviewed before people talk about them in public,” said Mr. Greene, a senior research fellow who works at the Davie, Fla., branch of the New York City-based Manhattan Institute. “I don’t think that’s a reasonable standard.”

Another signer, David N. Figlio, an economist at the University of Florida, in Gainesville, said he viewed the ad similarly. He was not one of the researchers who later publicized unreviewed work.

Mr. Figlio said he signed the original ad to protest the prominent play the Times had given to research that had never been professionally vetted. When the same ad appeared in Education Week, he said, he removed his name because he thought its coverage of the study had been more balanced.

“I feel strongly no unrefereed study should get front-page, above-the-fold treatment,” he said.

Setting Clear Standards

But Mr. Cibulka of the University of Kentucky, who also signed the ad, says the responsibility belongs to researchers from the start. “The problem is not with journalists,” he said. “The problem is with the profession for having failed to promulgate some clear standards.”

Mr. Cibulka said a trend toward making premature research findings public began as education became more prominent in federal and state policymaking, and as think tanks sprang up to try to influence those policymaking decisions.

David C. Berliner, an education professor at Arizona State University, in Tempe, added that “everybody has to have news all the time, and you can get stuff out there if you can tap into that.”

Mr. Greene said that one reason he sends his reports directly to the news media is that it’s quicker than publishing in traditional journals, which can take up to two years.

“In that time, people who are not researchers are commenting all the time on the merits of certain policies, and they’re doing so in the absence of evidence,” he said. “If we say that researchers can’t comment with evidence until that research has been peer-reviewed, ... we’ll actually have worse public-policy decisionmaking.”

Such practices, in comparison, would be taboo in biomedical science, according to Joann Rodgers, the director of media relations for Johns Hopkins medical school. She said her office won’t issue a press release on a researcher’s study unless it’s been accepted for publication in a peer-reviewed journal. She said medical school researchers respect those guidelines because the penalty for violating them is heavy. Journals such as Nature, The New England Journal of Medicine, and Science won’t accept a study if its findings have appeared elsewhere first, she said.

Peer-Review Critique

Even so, many researchers say peer review is never a guarantee of quality in any field.

“Not only do some lower-quality studies get published when the peer-review process is not rigorous enough, some important and high-quality studies may not get published because there is not a suitable outlet for their topic, methodology, or length,” Linda Darling-Hammond, the Stanford University education professor whose teacher-certification study was criticized last month, said in an e-mail interview.

“So any discussion of barriers to making research results public raises the possibility of staunching the free flow of ideas that is necessary for knowledge to move forward,” she added in her e-mail.

Ms. Darling-Hammond issued a press release on her study after circulating it among colleagues and then presenting it in Montreal last month at the annual meeting of the American Educational Research Association. (“Study Sees Positive Effects of Teacher Certification,” April 27, 2005.)

The Harvard University researchers who were criticized last month, Paul E. Peterson and Martin R. West, followed the same procedure when they presented their school accountability study at a British economics conference.

Felice J. Levine, the executive director of the 22,000-member, Washington-based AERA, said it’s important for researchers to be able to present their findings at public professional conferences because that’s one way to gather feedback from colleagues on how to improve the work.

Conference presentations do undergo some review in order to land a spot on the AERA agenda, she pointed out. But it’s not quite the same as undergoing a journal review, which typically involves a careful reading by at least three “blind” reviewers—colleagues, in other words, who have been chosen by the editor and whose identity is unknown to the author.

Perhaps what the profession needs, some education researchers said last week, is a review mechanism in the middle. One model, they said, might be the National Bureau of Education Research, which posts “working papers” on its Web site for colleagues’ reaction.

Mr. Cibulka suggested creating a national panel that could set standards for when to publicize research.

In retrospect, said Mr. Figlio of the University of Florida, “I think the charter school dust-up may have done us all a big favor in terms of thinking about how do we balance readiness and timeliness.”

Related Tags:

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Budget & Finance Webinar
Innovative Funding Models: A Deep Dive into Public-Private Partnerships
Discover how innovative funding models drive educational projects forward. Join us for insights into effective PPP implementation.
Content provided by Follett Learning
Budget & Finance Webinar Staffing Schools After ESSER: What School and District Leaders Need to Know
Join our newsroom for insights on investing in critical student support positions as pandemic funds expire.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Achievement Webinar
How can districts build sustainable tutoring models before the money runs out?
District leaders, low on funds, must decide: broad support for all or deep interventions for few? Let's discuss maximizing tutoring resources.
Content provided by Varsity Tutors for Schools

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Standards Opinion Did I Accurately Guess the Fate of the Common Core? You Be the Judge
In 2012, I imagined what the Common Core would be like in a decade. Now, readers can compare this imagined “future” to reality.
7 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Standards Minnesota Teachers, Parents Criticize ‘Awkward’ Tribal References in Proposed Math Standards
Teachers were somewhat more supportive of the tribal references than were parents, school board members, and school administrators.
Josh Verges, Pioneer Press
6 min read
Representatives from St. Paul Public Schools (SPPS) administration, faculty and staff, students, and the Indigenous community raise a healing pole in a ceremony at the SPPS headquarters in St. Paul, Monday, Oct. 11, 2021.
Representatives from St. Paul Public Schools (SPPS) administration, faculty and staff, students, and the Indigenous community raise a healing pole in a ceremony at the SPPS headquarters in St. Paul, Monday, Oct. 11, 2021.
Scott Takushi/Pioneer Press via TNS
Standards Social Studies Standards Spark Fierce Debate in N.C.
Advocates say the new standards are more inclusive because they give more attention to the perspectives of historically marginalized groups.
T. Keung Hui, The News & Observer (Raleigh, N.C.)
6 min read
Illustration.
Kubkoo/iStock/Getty
Standards Opinion How the Failure of the Common Core Looked From the Ground
Steve Peha shares insights from his on-site professional-development work about why the common core failed, in a guest letter to Rick Hess.
4 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty