Researchers themselves could be under the microscope for virtually the first time under new evaluation guidelines proposed by the Department of Education.
The guidelines call for peer-review panels to appraise all research grants awarded by the department--from the smallest purchase order to the largest federal research center--at least twice after they are approved. One review would come once the project was under way; a second would come when the work ended.
Currently, projects are evaluated by outside panels only once--when the grant is first awarded.
The change signals the end of a five-year effort to improve the quality of the research funded by the department. Congress, complaining that much of federally supported education research was fragmented, disconnected from the classroom, and politically driven, ordered the department in 1994 to reshape its major research arm, the office of educational research and improvement.
“One of the weaknesses of OERI is that they’ve never consolidated or reported what their contributions are,” said Gerald E. Sroufe, the lobbyist for the American Educational Research Association, which has its headquarters here.
He said the new standards would give researchers a chance to showcase their work as well as tighten the department’s control over quality.
The standards, published Feb. 24 in the Federal Register, would:
- Require at least two peer evaluations for grants of $5 million or less, and possibly one more interim review for larger awards;
- Set down guidelines for choosing peer reviewers; and
- Outline procedures for peer reviews and the criteria on which reviewers must base evaluations.
Process or Product?
“The interim assessments are really going to be the most important ones because that’s where we can make a difference in the success of the project,” said Margo Anderson, an OERI program analyst who helped draft the standards.
The proposed guidelines are the third set written by the department in its long drive to raise the standards for the research it supports. Earlier guidelines set down the operating criteria for the outside panels that initially assess grant proposals and for selecting “exemplary and promising” educational programs to share with educators across the country. (“Department Proposes Standards for Awarding Research Grants,” June 21, 1995.)
While several members of the research community said all of the new and proposed standards were a step in the right direction, some questioned last week whether the guidelines went far enough.
“A whole lot of this is process: Did you do what you said you would do?” said David Johnson, the executive director of the Federation of Behavioral, Psychological, and Cognitive Sciences, which is based here. “What you really want out of OERI is to have an impact on educational improvement.”
“You can’t answer about either impact or utility that close to when the work is being done,” he added.
Panel Criteria Questioned
In comparison, he noted, the National Science Foundation, which is also overhauling its procedures for evaluating grants, is considering guidelines that emphasize looking retrospectively at the national significance and the contributions of a research effort.
The OERI criteria for who sits on the peer-review panel may also prove contentious. As written now, anyone--including Education Department staff members--with knowledge or expertise in the area of the activities being reviewed can serve on a panel.
Some researchers questioned whether staff members would be able to offer independent, dispassionate views on the success or failure of department-financed projects. But others welcomed such staff input.
“I would hope the opinions and interpretations of the staff who’ve been following our work closely would also be factored in to the evaluations--whether that would come as part of the process itself or later on,” said Susan H. Fuhrman, the director of the Consortium for Policy Research in Education, based at the University of Pennsylvania, and dean of the university’s graduate school of education.
Including department staff members on review panels may also be a way of keeping the review process from becoming too unwieldy. With hundreds of projects to judge each year, drawing enough reviewers to fill all the panels needed could prove difficult, Ms. Anderson said.
The deadline for reaction to the proposed standards is April 27.
