Special Report
Education Funding

Peer Reviewers Winnow Race to Top Hopefuls

By Michele McNeil — February 26, 2010 7 min read

In the competition for $4 billion in Race to the Top grants, states have made their best pitches, a secret jury has debated and scored their applications—and now U.S. Secretary of Education Arne Duncan must decide who’s good enough to make it to the final round.

Mr. Duncan’s announcement of the Race to the Top Fund finalists, which is expected as early as this week, caps a dash by the Department of Education to recruit, vet, and train peer reviewers who wield tremendous power in determining who will win this high-stakes education reform competition.

In this first round of competition, all but 10 states applied for awards financed by the economic-stimulus package that will range from an estimated $350 million to $700 million—badly needed money that would help cash-strapped states.

Once the first-round awards are made in April, the department will publicize the scores and comments for each applicant from the 49 peer reviewers who have been given responsibility for judging 30 criteria that measure a state’s commitment to improving teacher quality, low-performing schools, data systems, and academic standards.

Despite all that influence, the ultimate power rests with Mr. Duncan, who will have the final say in awarding those cash prizes. But if the secretary deviates from the highest scores in awarding the Race to the Top grants, department officials have said he pledges to justify his reasons.

Inside the Peer Review Process

Each of 49 peer reviewers evaluated four or five state applications, which were randomly assigned. Comments and scores were due February 8, 2010.

23rtt c1s

IMAGES: iStockphoto.com/auras
SOURCES: U.S. Department of Education; Education Week

Mr. Duncan also has sole discretion in choosing how many finalists will come to Washington the week of March 15 to make presentations to the peer reviewers.

The purpose of the presentations, said Joanne Weiss, the department’s Race to the Top director, is not only to verify and clarify what’s in the applications, but also to ensure that those who must implement the plan know and understand what’s been promised.

“The real issue is, we’re giving away hundreds of millions of dollars, and we don’t want to just rely on what you wrote on a piece of paper,” Ms. Weiss said. “We want to look you in the eye.”

Looking for Conflicts

In August, the Education Department put out a call for peer reviewers to judge the applications and drew 1,500 applicants. They will pull from this pool for the second round of the competition, which starts in June when applications are due and ends in September, when the final dollar is awarded.

Vetting the list for potential conflicts of interest was no easy task. Any nominee who works for a state or district was automatically eliminated. So was anyone who helped out in any way on a state’s application, or who has a financial interest in the outcome of the competition.

Lawyers and ethics officers from the department did most of that work, although four career and four political staff members also reviewed the applicants to make sure they had expertise in the four school improvement areas emphasized in Race to the Top.

The department also hired a contractor to do a public-records search—even a Google search—into nominees to uncover information not previously disclosed. The department was also on the lookout for anyone who might have “significant identification with a specific pedagogical or philosophical viewpoint that might give an unfair advantage or disadvantage to an applicant,” according to department documents.

“It’s wasn’t like, ‘Do you have a belief?’ ” Ms. Weiss said. “It was more about, ‘Do you have an agenda?’ ”

Of the 49 peer reviewers, the department granted 12 waivers for “indirect” conflicts of interest, most commonly that a reviewer had worked for a state or school district within the past 12 months.

The reviewers receive $5,000 for what amounts to hundreds of hours of work that will span at least two months and involve tedious reading of applications that can exceed 1,000 pages each, counting appendices.

Though the department released basic demographic information on the reviewers—most, for example, have doctoral degrees and at least a dozen served on a state board of education or as a local or state school superintendent—federal officials have caught criticism for keeping their identities secret.

One reason the reviewers’ names should be made public is because of Mr. Duncan’s pledge to run a transparent competition, said Frederick M. Hess, the director of education policy studies for the American Enterprise Institute and a blogger hosted by Education Week. “We’re talking about an extraordinary sum of cash in play, and that’s why the identities of the judges are so important.”

In fact, the peer reviewers had to sign confidentiality agreements declaring they won’t disclose their identities or talk to the press—in effect, a gag order that applies until the department lifts it. Department officials have said they will make the identities public once the winners are announced, although it does not intend to reveal which reviewers scored which application.

Process So Far

In the course of the judging process, each peer reviewer received four or five applications to score; they were not allowed to judge applications from the states where they live. Then, the reviewers came together in Washington to form five-member panels to discuss and debate each application.

Since the reviewers rotate among panels, forming a unique one for each state, the department hopes to smooth out any scoring variances that might occur with different groups reviewing different applications.

Peer reviewers could refine their scores and comments based on their panel discussions, and the five scores were averaged to arrive at a final score. Ms. Weiss said any outliers whose grades were particularly low or high compared with the other panelists were not pressured to come to consensus.

Throughout the process, peer reviewers were instructed to do no independent research and to rely entirely on what’s written in the application.

Checks and balances were built into the system to fact-check the states’ applications. Ms. Weiss said, for example, that making the applications public, including the scores, comments, and video from the finalists’ presentations in Washington, will help ensure accountability.

Although the department has established a detailed, 500-point scoring guide, peer reviewers’ judgment will still ultimately come into play. For example, a peer reviewer will have to decide how to weigh an overly ambitious application, with a price tag that far exceeds the department’s nonbinding award estimates, against a less ambitious plan that stays within those estimates. “It’s entirely the peer reviewers’ judgment,” Ms. Weiss said. “You will see the reviewers made comments such as, ‘This is underfunded. Or this is overfunded.’ ”

Wild Card

Such judgment is the wild card in the competition, said Michael J. Petrilli, a vice president of the Thomas B. Fordham Institute, a think tank in Washington.

“What this comes down to is people. And invariably there are going to be judgment calls being made even as prescriptive as the criteria are,” said Mr. Petrilli, who worked with some peer-reviewed grant programs while at the department under President George W. Bush. “That’s when policy leanings are going to come into play.”

Still, the peer-review system was probably the best choice for the department in judging Race to the Top, said Mark Schneider, the vice president for education, human development, and the workforce program at the American Institutes for Research.

“Peer review is probably the best system we’ve come up with for judging products on merit,” said Mr. Schneider, a former commissioner of the National Center for Education Statistics. “If it’s done right, many people, especially losers, are unhappy but it does have a justification and a long history of support.”

There’s been less clarity to how Secretary Duncan and his staff will decide how many finalists will be selected—and how many awards will be given out.

The department has said that the applications will be ranked in order by score, with the top scorers getting money, until the money runs out. And Mr. Duncan has promised to be tough in setting a high bar for states to win.

Ms. Weiss said she couldn’t talk specifically about the process the department would use in determining the number of finalists or awards. But factors will include whether there are “natural breaks” that separate the superior applications from the not-so-good—and how much money is involved in awarding the grants.

One example: If a few big states with large grants are top scorers, there might be fewer finalists and fewer winners in the first round because the department has promised to leave a sizable pot of money for Phase 2.

And what will the losers get?

The same feedback as the winners, an average of seven to 10 pages of comments from each reviewer, which will serve as a roadmap for Phase 2.

Coverage of the American Recovery and Reinvestment Act is supported in part by grants from the William and Flora Hewlett Foundation, at www.hewlett.org and the Charles Stewart Mott Foundation, at www.mott.org.
A version of this article appeared in the March 03, 2010 edition of Education Week as Reviewers Winnow Race to Top Hopefuls


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
How Principals Can Support Student Well-Being During COVID
Join this webinar for tips on how to support and prioritize student health and wellbeing during COVID.
Content provided by Unruly Studios
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Professional Development Webinar
Building Leadership Excellence Through Instructional Coaching
Join this webinar for a discussion on instructional coaching and ways you can link your implement or build on your program.
Content provided by Whetstone Education/SchoolMint
Teaching Webinar Tips for Better Hybrid Learning: Ask the Experts What Works
Register and ask your questions about hybrid learning to our expert panel.

EdWeek Top School Jobs

CCLC Program Site Director
Thornton, CO, US
Adams 12 Five Star Schools
CCLC Program Site Director
Thornton, CO, US
Adams 12 Five Star Schools
Marketing Coordinator
Portland, OR, US
Northwest Evaluation Association
Sr Project Manager, Marketing (Temporary)
Portland, OR, US
Northwest Evaluation Association

Read Next

Education Funding Concern About Unspent COVID-19 School Aid Continues as Congress Moves Toward More Relief
A congressional analysis has spurred discontent about how fast money will be spent, but some warn against over-simplifying the situation.
5 min read
Thermometers, gloves, and cleaning swabs sit on a table at the entrance to the Frederickson KinderCare daycare center, in Tacoma, Wash on May 27, 2020.
Thermometers, gloves, and cleaning swabs sit on a table at the entrance to the Frederickson KinderCare daycare center, in Tacoma, Wash on May 27, 2020. As a precaution against the spread of the coronavirus, workers and children have their temperatures checked every day before they enter the building.
Ted S. Warren/AP
Education Funding Opinion What's It Take for Philanthropy to Help Rural Schools?
Place-based philanthropy has enormous potential to revitalize rural communities. The trick is, it’s tough to get this sort of approach right.
4 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Education Funding Summer School, Extended Learning a Priority in $129 Billion COVID-19 Relief Bill
The reconciliation bill from House Democrats is an early attempt at what's shaping up to be a new federal aid package for education.
5 min read
A staff member holds the door open for kids on the first day of school at Goodwin Frazier Elementary School in New Braunfels, Texas on Tuesday, Aug. 25, 2020.
A staff member holds the door open for kids on the first day of school at Goodwin Frazier Elementary School in New Braunfels, Texas on Tuesday, Aug. 25, 2020.
Mikala Compton/Herald-Zeitung via AP
Education Funding Details of Biden's Education Relief Pitch Prioritize Smaller Classes, Avoiding Layoffs
The administration's breakdown of COVID-19 needs also includes $50 billion for social distancing and $29 billion for learning recovery.
5 min read
Image of the White House seal
Bet Noire/Getty