School & District Management

Panel Urges Tighter Review of Research-Grant Proposals

By Debra Viadero — March 24, 1999 3 min read

If the Department of Education is going to raise the credibility of the research it supports, a new report concludes, it must bolster its system for outside review of grant proposals.

For More Information

Free copies of “Strengthening the Standards: Recommendations for OERI Peer Review” will be available after April 1 from the National Educational Research Policy and Priorities Board at (202) 208-0692 or by e-mail at eve_bither@ed.gov.

At the direction of Congress, the department four years ago tightened guidelines governing the peer-review panels that evaluate proposals for agency funding. But the report, commissioned by an advisory board to the department, suggests that the changes didn’t go far enough.

Some of the problems cited in the report include:

Instances of evaluation panels whose members had almost no expertise in research methodology; At least one reviewer who had not read the proposals before a review meeting; Complaints from applicants that reviewers had misunderstood or misstated their proposals; and Cursory written reviews of proposals that gave researchers little or no substantive feedback.

The lack of experienced researchers on some of the panels is the most serious shortcoming in the department’s peer-review practices, concludes the study released last month. For example, in a 1997 competition for “field initiated” grants--which are essentially funding for researchers’ pet projects--12 of the 35 reviewers who sat on peer panels had no research experience or publications on their r‚sum‚s.

Expertise Pays Off

“They’re not peers if they’re not engaged in research,” said Carl F. Kaestle, a prominent Brown University researcher who sat on the panel that directed the study. Two private research firms--August and Associates in Bethesda, Md., and Lana D. Muraskin of Washington--conducted the study.

“The standards say all peer reviewers should have knowledge of research as well as what’s going on in the field,” Mr. Kaestle said.

Technical expertise is key, the report says, because it appears to lead to better-quality reviews. The three panels whose members had no research training produced no reviews that were rated “good” by the study team. The opposite was true for panels that included researchers working in the same fields as the applicants. Most of their reviews were rated “good,” and none was deemed “poor.”

Part of the problem has been that reviewers with the know-how called for in the guidelines are hard to find, the report points out. Besides knowing something about the subject area of the proposed research, reviewers need to have in-depth knowledge of research methods and of educational policy and practice. And department staff members, sometimes given only three weeks to recruit reviewers, have to mix and match panel experts.

One remedy to the problem, the panel concludes, might be to establish standing review panels for each of the department’s five national research institutes.

Changes Considered

“If you have sort of a one-shot, ad hoc meeting of people with all those pieces of knowledge, you run the risk of having people who are not really experts in any of those areas,” said Kenji Hakuta, the chairman of the National Educational Research Policy and Priorities Board, which commissioned the report. The congressionally mandated board endorsed the report at its January meeting. “With a standing panel, you have to meet at least a minimal level of expertise for everyone,” Mr. Hakuta added.

Proposed three-year terms for panel members, recommended in the report, would also offer more time for training reviewers.

Department officials are studying that idea, which may require congressional approval, as well as the report’s other recommendations.

But some of the changes called for are already being tested in the current competition for field-initiated studies. This year, study proposals will be screened first through a panel of researchers, who will assess their technical quality. The research designs making that cut will then go before a broader panel made up of practitioners and researchers.

“It’s something we would’ve done whether or not we had the peer-review report,” said C. Kent McGuire, the assistant secretary in charge of the department’s office of educational research and improvement. “But this report gives further credence to the need to do something about peer review in this place.”

Related Tags:

A version of this article appeared in the March 24, 1999 edition of Education Week as Panel Urges Tighter Review of Research-Grant Proposals

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Interactive Learning Best Practices: Creative Ways Interactive Displays Engage Students
Students and teachers alike struggle in our newly hybrid world where learning takes place partly on-site and partly online. Focus, engagement, and motivation have become big concerns in this transition. In this webinar, we will
Content provided by Samsung
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Educator-Driven EdTech Design: Help Shape the Future of Classroom Technology
Join us for a collaborative workshop where you will get a live demo of GoGuardian Teacher, including seamless new integrations with Google Classroom, and participate in an interactive design exercise building a feature based on
Content provided by GoGuardian
School & District Management Live Online Discussion A Seat at the Table With Education Week: What Did We Learn About Schooling Models This Year?
After a year of living with the pandemic, what schooling models might we turn to as we look ahead to improve the student learning experience? Could year-round schooling be one of them? What about online

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management 'You Can’t Follow CDC Guidelines': What Schools Really Look Like During COVID-19
All year, some teachers have said that enforcing precautions to slow the spread of the virus in classrooms can be nearly impossible.
13 min read
Guntown Middle School eighth graders walk the halls to their next class as others wait in their assigned spots against the wall before moving into their next class during the first day back to school for the Lee County District in Guntown, Miss on Aug. 6, 2020.
Eight graders walk the halls on the first day back to school in Guntown, Miss., on Aug. 6, 2020. Teachers in several states told Education Week that since the beginning of the school year, enforcing precautions such as social distancing to slow the spread of the coronavirus has been nearly impossible.<br/>
Adam Robison/The Northeast Mississippi Daily Journal via AP
School & District Management Opinion School Reopening Requires More Than Just Following the Science
Educators can only “follow the science” so far. Professional expertise matters too, writes Susan Moore Johnson.
Susan Moore Johnson
5 min read
Illustration of school and bus
Getty
School & District Management Why Teacher Vaccinations Are So Hard to Track
Teachers can now get the COVID-19 vaccine, but there’s no way of knowing how many are currently inoculated against the virus.
6 min read
Image of a needle and vaccine bottle.
iStock/Getty
School & District Management Do Teachers Have to Disclose Their Vaccination Status? Experts Weigh In
Experts answer four pressing questions about teachers, privacy, and COVID-19 vaccines.
3 min read
Vaccine record.
Bill Oxford/iStock/Getty