Special Report
Education Funding

Peer Reviewers Winnow Race to Top Hopefuls

By Michele McNeil — February 26, 2010 7 min read

In the competition for $4 billion in Race to the Top grants, states have made their best pitches, a secret jury has debated and scored their applications—and now U.S. Secretary of Education Arne Duncan must decide who’s good enough to make it to the final round.

Mr. Duncan’s announcement of the Race to the Top Fund finalists, which is expected as early as this week, caps a dash by the Department of Education to recruit, vet, and train peer reviewers who wield tremendous power in determining who will win this high-stakes education reform competition.

In this first round of competition, all but 10 states applied for awards financed by the economic-stimulus package that will range from an estimated $350 million to $700 million—badly needed money that would help cash-strapped states.

Once the first-round awards are made in April, the department will publicize the scores and comments for each applicant from the 49 peer reviewers who have been given responsibility for judging 30 criteria that measure a state’s commitment to improving teacher quality, low-performing schools, data systems, and academic standards.

Despite all that influence, the ultimate power rests with Mr. Duncan, who will have the final say in awarding those cash prizes. But if the secretary deviates from the highest scores in awarding the Race to the Top grants, department officials have said he pledges to justify his reasons.

Inside the Peer Review Process

Each of 49 peer reviewers evaluated four or five state applications, which were randomly assigned. Comments and scores were due February 8, 2010.

BRIC ARCHIVE

IMAGES: iStockphoto.com/auras
SOURCES: U.S. Department of Education; Education Week

Mr. Duncan also has sole discretion in choosing how many finalists will come to Washington the week of March 15 to make presentations to the peer reviewers.

The purpose of the presentations, said Joanne Weiss, the department’s Race to the Top director, is not only to verify and clarify what’s in the applications, but also to ensure that those who must implement the plan know and understand what’s been promised.

“The real issue is, we’re giving away hundreds of millions of dollars, and we don’t want to just rely on what you wrote on a piece of paper,” Ms. Weiss said. “We want to look you in the eye.”

Looking for Conflicts

In August, the Education Department put out a call for peer reviewers to judge the applications and drew 1,500 applicants. They will pull from this pool for the second round of the competition, which starts in June when applications are due and ends in September, when the final dollar is awarded.

Vetting the list for potential conflicts of interest was no easy task. Any nominee who works for a state or district was automatically eliminated. So was anyone who helped out in any way on a state’s application, or who has a financial interest in the outcome of the competition.

Lawyers and ethics officers from the department did most of that work, although four career and four political staff members also reviewed the applicants to make sure they had expertise in the four school improvement areas emphasized in Race to the Top.

The department also hired a contractor to do a public-records search—even a Google search—into nominees to uncover information not previously disclosed. The department was also on the lookout for anyone who might have “significant identification with a specific pedagogical or philosophical viewpoint that might give an unfair advantage or disadvantage to an applicant,” according to department documents.

“It’s wasn’t like, ‘Do you have a belief?’ ” Ms. Weiss said. “It was more about, ‘Do you have an agenda?’ ”

Of the 49 peer reviewers, the department granted 12 waivers for “indirect” conflicts of interest, most commonly that a reviewer had worked for a state or school district within the past 12 months.

The reviewers receive $5,000 for what amounts to hundreds of hours of work that will span at least two months and involve tedious reading of applications that can exceed 1,000 pages each, counting appendices.

Though the department released basic demographic information on the reviewers—most, for example, have doctoral degrees and at least a dozen served on a state board of education or as a local or state school superintendent—federal officials have caught criticism for keeping their identities secret.

One reason the reviewers’ names should be made public is because of Mr. Duncan’s pledge to run a transparent competition, said Frederick M. Hess, the director of education policy studies for the American Enterprise Institute and a blogger hosted by Education Week. “We’re talking about an extraordinary sum of cash in play, and that’s why the identities of the judges are so important.”

In fact, the peer reviewers had to sign confidentiality agreements declaring they won’t disclose their identities or talk to the press—in effect, a gag order that applies until the department lifts it. Department officials have said they will make the identities public once the winners are announced, although it does not intend to reveal which reviewers scored which application.

Process So Far

In the course of the judging process, each peer reviewer received four or five applications to score; they were not allowed to judge applications from the states where they live. Then, the reviewers came together in Washington to form five-member panels to discuss and debate each application.

Since the reviewers rotate among panels, forming a unique one for each state, the department hopes to smooth out any scoring variances that might occur with different groups reviewing different applications.

Peer reviewers could refine their scores and comments based on their panel discussions, and the five scores were averaged to arrive at a final score. Ms. Weiss said any outliers whose grades were particularly low or high compared with the other panelists were not pressured to come to consensus.

Throughout the process, peer reviewers were instructed to do no independent research and to rely entirely on what’s written in the application.

Checks and balances were built into the system to fact-check the states’ applications. Ms. Weiss said, for example, that making the applications public, including the scores, comments, and video from the finalists’ presentations in Washington, will help ensure accountability.

Although the department has established a detailed, 500-point scoring guide, peer reviewers’ judgment will still ultimately come into play. For example, a peer reviewer will have to decide how to weigh an overly ambitious application, with a price tag that far exceeds the department’s nonbinding award estimates, against a less ambitious plan that stays within those estimates. “It’s entirely the peer reviewers’ judgment,” Ms. Weiss said. “You will see the reviewers made comments such as, ‘This is underfunded. Or this is overfunded.’ ”

Wild Card

Such judgment is the wild card in the competition, said Michael J. Petrilli, a vice president of the Thomas B. Fordham Institute, a think tank in Washington.

“What this comes down to is people. And invariably there are going to be judgment calls being made even as prescriptive as the criteria are,” said Mr. Petrilli, who worked with some peer-reviewed grant programs while at the department under President George W. Bush. “That’s when policy leanings are going to come into play.”

Still, the peer-review system was probably the best choice for the department in judging Race to the Top, said Mark Schneider, the vice president for education, human development, and the workforce program at the American Institutes for Research.

“Peer review is probably the best system we’ve come up with for judging products on merit,” said Mr. Schneider, a former commissioner of the National Center for Education Statistics. “If it’s done right, many people, especially losers, are unhappy but it does have a justification and a long history of support.”

There’s been less clarity to how Secretary Duncan and his staff will decide how many finalists will be selected—and how many awards will be given out.

The department has said that the applications will be ranked in order by score, with the top scorers getting money, until the money runs out. And Mr. Duncan has promised to be tough in setting a high bar for states to win.

Ms. Weiss said she couldn’t talk specifically about the process the department would use in determining the number of finalists or awards. But factors will include whether there are “natural breaks” that separate the superior applications from the not-so-good—and how much money is involved in awarding the grants.

One example: If a few big states with large grants are top scorers, there might be fewer finalists and fewer winners in the first round because the department has promised to leave a sizable pot of money for Phase 2.

And what will the losers get?

The same feedback as the winners, an average of seven to 10 pages of comments from each reviewer, which will serve as a roadmap for Phase 2.

Coverage of the American Recovery and Reinvestment Act is supported in part by grants from the William and Flora Hewlett Foundation, at www.hewlett.org and the Charles Stewart Mott Foundation, at www.mott.org.
A version of this article appeared in the March 03, 2010 edition of Education Week as Reviewers Winnow Race to Top Hopefuls

Events

Jobs The EdWeek Top School Jobs Virtual Career Fair
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Mathematics Webinar
Engaging Young Students to Accelerate Math Learning
Join learning scientists and inspiring district leaders, for a timely panel discussion addressing a school district’s approach to doubling and tripling Math gains during Covid. What started as a goal to address learning gaps in
Content provided by Age of Learning & Digital Promise, Harlingen CISD
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How to Power Your Curriculum With Digital Books
Register for this can’t miss session looking at best practices for utilizing digital books to support their curriculum.
Content provided by OverDrive

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Funding To Get Billions in COVID-19 Aid, States Pledge Focus on Mental Health, Learning Recovery
Twenty-eight states had submitted plans to the Education Department as of mid-June to access $41 billion from the American Rescue Plan.
4 min read
Illustration of money floating in a life preserver.
ISerg/iStock/Getty
Education Funding Some in Congress Fear State Budget Decisions May Undercut COVID-19 Education Relief
A dispute in Wisconsin over coronavirus relief underscores how technical issues and politics are affecting education spending decisions.
4 min read
Image shows an illustration of money providing relief against coronavirus.
DigitalVision Vectors/iStock/Getty
Education Funding There Are Big Funding Gaps Affecting High-Poverty Schools. Can Biden Close Them?
Hurdles lie ahead for a $20 billion bid to create "Title I equity grants" to address long-standing funding inequities.
9 min read
President Joe Biden talks about the May jobs report from the Rehoboth Beach Convention Center in Rehoboth Beach, Del., Friday, June 4, 2021.
President Joe Biden made boosting Title I for disadvantaged students a key part of his education platform on the campaign trail.
Susan Walsh/AP
Education Funding Education Department Issues Directive on Shielding Students in Poverty From Funding Cuts
The agency released the "maintenance of equity" guidance on COVID-19 relief as part of a public-relations blitz on equity amid the pandemic.
5 min read
Image of a $100 dollar bill that is cut into blocks for distribution.
E+/Getty