Federal

Scholars Spar Over Research Methods Used to Evaluate Charters

By Debra Viadero — October 08, 2009 4 min read
  • Save to favorites
  • Print

Includes updates and/or revisions.

The authors of a recent national study that found students in regular public schools outperforming their charter school peers are rebutting criticism that their research suffered from a “serious mathematical mistake” that negatively skewed the results.

In a memorandum posted online last week, researchers from the Center for Research on Education Outcomes, or CREDO, at Stanford University responded to criticism made last month by their Stanford colleague Caroline M. Hoxby. Ms. Hoxby, an economics professor, issued a memo critiquing the CREDO study in tandem with results from her own study of charter schools in New York City.

That study showed that charter schools in the city were having the opposite effect on their students’ achievement as the CREDO researchers found. Compared with their public school counterparts, Ms. Hoxby found, students in charter schools made more progress in closing achievement gaps with better-off peers in the suburbs. (“N.Y.C. Charters Found to Close Gaps,” Sept. 22, 2009.)

“She claims that we modeled charter schools in a particular way and that was inaccurate,” Margaret E. Raymond, CREDO’s director, said of Ms. Hoxby’s critique of the statistical analysis. “And her proof was based on her inaccurate specification.”

Ms. Raymond said Ms. Hoxby’s critique also failed to consider the many factors that the CREDO study took into account in its analysis.

Stakes High

“Imagine that we’re going to compare height and shoe size, and that we’re going to do it for all the people in the office,” Ms. Raymond said. “Those two things might actually correlate in a particular way because they’re only two factors. But if we’re going to predict shoe size based on many more factors, like age or parents’ height or parents’ shoe sizes, that’s a much more precise way of estimating.”

Criticism and Rebuttal

With charter schools a high-profile policy issue, researchers who came to opposite conclusions about their effect on student achievement are debating their analyses.

“She claims that we modeled charter schools in a particular way, and that was inaccurate. And her proof was based on her inaccurate speciication.”

Margaret E. Raymond
Director, Center for Research on Education Outcomes
Stanford University

“They assert that the proof proceeds in a way that it does not. Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

Caroline M. Hoxby
Professor of Economics
Stanford University

The CREDO researchers also argue in their memorandum that Ms. Hoxby’s estimates of the magnitude of the bias in their results was faulty.

The researchers’ arguments failed, however, to persuade Ms. Hoxby. In an e-mail response to Education Week, she said CREDO’s analysis contained “logically incorrect assertions.”

“They assert that the proof proceeds in a way that it does not,” Ms. Hoxby wrote. “Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

If the debate is taking on a bitter tone, it’s because the stakes are high. U.S. Secretary of Education Arne Duncan has been pressuring states to remove caps on charter schools in order to qualify for some of the $100 billion in economic-stimulus funds targeted to education. But opposition to the schools, which are public schools that operate largely free of the usual regulations, continues to come from teachers’ union officials and other education leaders.

“I don’t think the field of education research or policymakers are well served by scholars going back and forth with dueling memos, without peer review and without ample time to think it through,” said Sean F. Reardon, a colleague of both scholars at Stanford. “But I don’t think either side got it right,” he added.

An associate professor of education and sociology, Mr. Reardon is reviewing the New York City study. But he said he’s also not yet convinced that the CREDO study “does enough to ensure that it’s entirely free of bias.”

“Like any nonexperimental study,” he added, “it relies on some strong assumptions that are hard to justify.”

Part of the problem, he said, is that neither study provides enough statistical detail about the data and the methods used to help reviewers settle the debate.

Also, neither study has been published yet in a peer-reviewed journal. However, an earlier version of Ms. Hoxby’s report is posted as a working paper on the Web site of the National Bureau of Economic Research, which provides a forum for economists seeking feedback from the field on their work. Ms. Hoxby said her paper is also under consideration for publication by two journals.

The CREDO study was reviewed by four outside experts before its release this past June, according to a publicist. (“Study Casts Doubt on Charter School Results,” June 15, 2009)

The memo last week from CREDO suggests that it’s not surprising that the two studies should yield different results, given that a central finding of the CREDO study, which encompasses 15 states and the District of Columbia, is that charter school performance varies considerably from school to school.

“Some communities and states have gotten the policy right, and are able to demonstrate positive charter school results,” the CREDO researchers write. “The recent results for New York City indicate it, too, has a focused and effective charter policy.

“Our hope is that we can get beyond the distractions of this technical discussion,” the rebuttal adds, “and return to the larger policy issues.”

A version of this article appeared in the October 14, 2009 edition of Education Week as Charter School Researchers Spar Over Analyses

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Education Dept. Sees Small Cut in Funding Package That Averted Government Shutdown
The Education Department will see a reduction even as the funding package provides for small increases to key K-12 programs.
3 min read
President Joe Biden delivers a speech about healthcare at an event in Raleigh, N.C., on March 26, 2024.
President Joe Biden delivers a speech about health care at an event in Raleigh, N.C., on March 26. Biden signed a funding package into law over the weekend that keeps the federal government open through September but includes a slight decrease in the Education Department's budget.
Matt Kelley/AP
Federal What Would Happen to K-12 in a 2nd Trump Term? A Detailed Policy Agenda Offers Clues
A conservative policy agenda could offer the clearest view yet of K-12 education in a second Trump term.
8 min read
Republican presidential candidate and former President Donald Trump speaks at a campaign rally, March 9, 2024, in Rome Ga.
Former President Donald Trump speaks at a campaign rally, March 9, 2024, in Rome, Ga. Allies of the former president have assembled a detailed policy agenda for every corner of the federal government with the idea that it would be ready for a conservative president to use at the start of a new term next year.
Mike Stewart/AP
Federal Opinion Reading Scores Have Plunged Since the Pandemic. What This Senator Wants to Do About That
How can the nation support literacy efforts? The ranking Republican on the Senate education committee wants to hear from educators.
6 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
Federal Biden Calls for Teacher Pay Raises, Expanded Pre-K in State of the Union
President Joe Biden highlighted a number of his education priorities in a high-stakes speech as he seeks a second term.
5 min read
President Joe Biden delivers the State of the Union address to a joint session of Congress at the Capitol on March 7, 2024, in Washington.
President Joe Biden delivers the State of the Union address to a joint session of Congress at the Capitol on March 7, 2024, in Washington.
Shawn Thew/Pool via AP