Standards

Release of Unreviewed Studies Sparks Debate

By Debra Viadero — May 17, 2005 6 min read

Researchers from two of the nation’s most eminent universities, Harvard and Stanford, recently unveiled studies on different, but equally controversial, education issues. One was on school accountability policy, the other on teacher certification, but critics’ responses were the same.

They complained that the studies were “not quite ready for prime time” because they had not undergone the independent scrutiny that usually comes when studies are published in academic journals.

It’s a refrain that has become common lately as research on hot-button issues moves directly from the author’s computer printer to a press release. The criticism has some scholars wondering whether education researchers need to do a better job of policing themselves.

“I think there have been abuses on all sides,” said James. G. Cibulka, the dean of the college of education at the University of Kentucky, in Lexington. “As a profession, we really need to have some agreed-upon criteria for vetting studies before they make their way to the public.”

Whether the buzz about the need for giving studies some level of professional review before they hit the public arena is new, or even limited to education research, is an open question.

Ready for the Public?

Studies generally undergo an evaluation process before they are accepted for publication in an academic journal, presented at a conference, or released by a think tank.

Peer review for articles published in a professional journal typically involves three or more reviewers who critique an author’s work in writing. The reviewers are “blind,” meaning that they are chosen by the journal editor and that the author does not know their identities.

Peer review for presentations at a professional conference is a less-rigorous procedure. Reviewers often accept or reject a proposed presentation on the basis of an abstract or a three- or four-page paper.

Peer review at a nonpartisan think tank, such as the New York City-based MDRC or the Washington-based American Institutes for Research, takes place mostly in-house as think tank researchers review one another’s work. In the later stages of its three-tiered review process, MDRC also enlists board members, outside experts in the topic under study, and participants from the study sites. For the Manhattan Institute, a think tank widely seen to support conservative perspectives, researcher Jay P. Greene said he typically circulates his papers to several outside colleagues before releasing them to the public.

“One of the traditions of the education research enterprise has been the high frequency of advocacy research,” said Grover J. “Russ” Whitehurst, the director of the U.S. Department of Education’s Institute of Education Sciences. “I would be skeptical that there’s been an uptick on this.”

Others disagree. They say the debate over the need for tighter peer-review mechanisms sharpened last summer after the American Federation of Teachers produced an unreviewed study suggesting that some students in public elementary schools outperformed their counterparts in the same grades in charter schools on national tests of reading and mathematics. (“AFT Charter School Study Sparks Heated National Debate,” Sept. 1, 2004.)

When the AFT study’s findings became the subject of the lead story in The New York Times on Aug. 17, supporters of charter schools mounted a counterattack that included editorials, letters to the editor, and a full-page advertisement in national newspapers. The ad, signed by 31 academics, mostly criticized the study on methodological grounds, but it also called for closer scrutiny of research before findings percolate into the mainstream media. It said that “such studies need to be vetted by independent scholars, as is commonly done in coverage of research on the biological and physical sciences.”

Since then, what has irked some other researchers is that three of the signers went on to do just what the teachers’ union had done: They took research findings to the press before the studies had undergone a formal peer-review process. That drew charges of hypocrisy from some of their research colleagues.

Jay P. Greene, one of the signers who was criticized, said he saw the ad as more of an admonition to journalists than to researchers.

“The ad never said all studies have to be peer-reviewed before people talk about them in public,” said Mr. Greene, a senior research fellow who works at the Davie, Fla., branch of the New York City-based Manhattan Institute. “I don’t think that’s a reasonable standard.”

Another signer, David N. Figlio, an economist at the University of Florida, in Gainesville, said he viewed the ad similarly. He was not one of the researchers who later publicized unreviewed work.

Mr. Figlio said he signed the original ad to protest the prominent play the Times had given to research that had never been professionally vetted. When the same ad appeared in Education Week, he said, he removed his name because he thought its coverage of the study had been more balanced.

“I feel strongly no unrefereed study should get front-page, above-the-fold treatment,” he said.

Setting Clear Standards

But Mr. Cibulka of the University of Kentucky, who also signed the ad, says the responsibility belongs to researchers from the start. “The problem is not with journalists,” he said. “The problem is with the profession for having failed to promulgate some clear standards.”

Mr. Cibulka said a trend toward making premature research findings public began as education became more prominent in federal and state policymaking, and as think tanks sprang up to try to influence those policymaking decisions.

David C. Berliner, an education professor at Arizona State University, in Tempe, added that “everybody has to have news all the time, and you can get stuff out there if you can tap into that.”

Mr. Greene said that one reason he sends his reports directly to the news media is that it’s quicker than publishing in traditional journals, which can take up to two years.

“In that time, people who are not researchers are commenting all the time on the merits of certain policies, and they’re doing so in the absence of evidence,” he said. “If we say that researchers can’t comment with evidence until that research has been peer-reviewed, ... we’ll actually have worse public-policy decisionmaking.”

Such practices, in comparison, would be taboo in biomedical science, according to Joann Rodgers, the director of media relations for Johns Hopkins medical school. She said her office won’t issue a press release on a researcher’s study unless it’s been accepted for publication in a peer-reviewed journal. She said medical school researchers respect those guidelines because the penalty for violating them is heavy. Journals such as Nature, The New England Journal of Medicine, and Science won’t accept a study if its findings have appeared elsewhere first, she said.

Peer-Review Critique

Even so, many researchers say peer review is never a guarantee of quality in any field.

“Not only do some lower-quality studies get published when the peer-review process is not rigorous enough, some important and high-quality studies may not get published because there is not a suitable outlet for their topic, methodology, or length,” Linda Darling-Hammond, the Stanford University education professor whose teacher-certification study was criticized last month, said in an e-mail interview.

“So any discussion of barriers to making research results public raises the possibility of staunching the free flow of ideas that is necessary for knowledge to move forward,” she added in her e-mail.

Ms. Darling-Hammond issued a press release on her study after circulating it among colleagues and then presenting it in Montreal last month at the annual meeting of the American Educational Research Association. (“Study Sees Positive Effects of Teacher Certification,” April 27, 2005.)

The Harvard University researchers who were criticized last month, Paul E. Peterson and Martin R. West, followed the same procedure when they presented their school accountability study at a British economics conference.

Felice J. Levine, the executive director of the 22,000-member, Washington-based AERA, said it’s important for researchers to be able to present their findings at public professional conferences because that’s one way to gather feedback from colleagues on how to improve the work.

Conference presentations do undergo some review in order to land a spot on the AERA agenda, she pointed out. But it’s not quite the same as undergoing a journal review, which typically involves a careful reading by at least three “blind” reviewers—colleagues, in other words, who have been chosen by the editor and whose identity is unknown to the author.

Perhaps what the profession needs, some education researchers said last week, is a review mechanism in the middle. One model, they said, might be the National Bureau of Education Research, which posts “working papers” on its Web site for colleagues’ reaction.

Mr. Cibulka suggested creating a national panel that could set standards for when to publicize research.

In retrospect, said Mr. Figlio of the University of Florida, “I think the charter school dust-up may have done us all a big favor in terms of thinking about how do we balance readiness and timeliness.”

Related Tags:

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Equity, Care and Connection: New SEL Tools and Practices to Support Students and Adults
As school districts plan to welcome students back into buildings for the upcoming school year, this is the perfect time to take a hard look at both our practices and our systems to build a
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Here to Stay – Pandemic Lessons for EdTech in Future Development
What technology is needed in a post pandemic district? Learn how changes in education will impact development of new technologies.
Content provided by AWS
School & District Management Live Online Discussion A Seat at the Table: Strategies & Tips for Complex Decision-Making
Schools are working through the most disruptive period in the history of modern education, facing a pandemic, economic problems, social justice issues, and rapid technological change all at once. But even after the pandemic ends,

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Standards Opinion Common Core Is a Meal Kit, Not a Nothingburger
Caroline Damon argues Rick Hess and Tom Loveless sold the common core short, claiming the issue was a matter of high-quality implementation.
5 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Standards How New Common Core Research Connects to Biden's Plans for Children and Families
A study of national test scores indicate the early phase of the Common Core State Standards did not help disadvantaged students.
5 min read
results 925693186 02
iStock/Getty
Standards Opinion After All That Commotion, Was the Common Core a Big Nothingburger?
The Common Core State Standards may not have had an impact on student outcomes, but they did make school improvement tougher and more ideological.
3 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Standards New Media Literacy Standards Aim to Combat 'Truth Decay'
The RAND Corporation has released a set of media literacy standards designed to help teach students to identify misinformation.
6 min read
Visual shows a young woman lying down and watching live news content on her mobile phone.
gorodenkoff/iStock/Getty Images Plus