If the Department of Education is going to raise the credibility of the research it supports, a new report concludes, it must bolster its system for outside review of grant proposals.
For More Information |
| Free copies of “Strengthening the Standards: Recommendations for OERI Peer Review” will be available after April 1 from the National Educational Research Policy and Priorities Board at (202) 208-0692 or by e-mail at eve_bither@ed.gov. |
At the direction of Congress, the department four years ago tightened guidelines governing the peer-review panels that evaluate proposals for agency funding. But the report, commissioned by an advisory board to the department, suggests that the changes didn’t go far enough.
Some of the problems cited in the report include:
Instances of evaluation panels whose members had almost no expertise in research methodology; At least one reviewer who had not read the proposals before a review meeting; Complaints from applicants that reviewers had misunderstood or misstated their proposals; and Cursory written reviews of proposals that gave researchers little or no substantive feedback.
The lack of experienced researchers on some of the panels is the most serious shortcoming in the department’s peer-review practices, concludes the study released last month. For example, in a 1997 competition for “field initiated” grants--which are essentially funding for researchers’ pet projects--12 of the 35 reviewers who sat on peer panels had no research experience or publications on their r‚sum‚s.
Expertise Pays Off
“They’re not peers if they’re not engaged in research,” said Carl F. Kaestle, a prominent Brown University researcher who sat on the panel that directed the study. Two private research firms--August and Associates in Bethesda, Md., and Lana D. Muraskin of Washington--conducted the study.
“The standards say all peer reviewers should have knowledge of research as well as what’s going on in the field,” Mr. Kaestle said.
Technical expertise is key, the report says, because it appears to lead to better-quality reviews. The three panels whose members had no research training produced no reviews that were rated “good” by the study team. The opposite was true for panels that included researchers working in the same fields as the applicants. Most of their reviews were rated “good,” and none was deemed “poor.”
Part of the problem has been that reviewers with the know-how called for in the guidelines are hard to find, the report points out. Besides knowing something about the subject area of the proposed research, reviewers need to have in-depth knowledge of research methods and of educational policy and practice. And department staff members, sometimes given only three weeks to recruit reviewers, have to mix and match panel experts.
One remedy to the problem, the panel concludes, might be to establish standing review panels for each of the department’s five national research institutes.
Changes Considered
“If you have sort of a one-shot, ad hoc meeting of people with all those pieces of knowledge, you run the risk of having people who are not really experts in any of those areas,” said Kenji Hakuta, the chairman of the National Educational Research Policy and Priorities Board, which commissioned the report. The congressionally mandated board endorsed the report at its January meeting. “With a standing panel, you have to meet at least a minimal level of expertise for everyone,” Mr. Hakuta added.
Proposed three-year terms for panel members, recommended in the report, would also offer more time for training reviewers.
Department officials are studying that idea, which may require congressional approval, as well as the report’s other recommendations.
But some of the changes called for are already being tested in the current competition for field-initiated studies. This year, study proposals will be screened first through a panel of researchers, who will assess their technical quality. The research designs making that cut will then go before a broader panel made up of practitioners and researchers.
“It’s something we would’ve done whether or not we had the peer-review report,” said C. Kent McGuire, the assistant secretary in charge of the department’s office of educational research and improvement. “But this report gives further credence to the need to do something about peer review in this place.”
