Federal

Scholars Spar Over Research Methods Used to Evaluate Charters

By Debra Viadero — October 08, 2009 4 min read
  • Save to favorites
  • Print

Includes updates and/or revisions.

The authors of a recent national study that found students in regular public schools outperforming their charter school peers are rebutting criticism that their research suffered from a “serious mathematical mistake” that negatively skewed the results.

In a memorandum posted online last week, researchers from the Center for Research on Education Outcomes, or CREDO, at Stanford University responded to criticism made last month by their Stanford colleague Caroline M. Hoxby. Ms. Hoxby, an economics professor, issued a memo critiquing the CREDO study in tandem with results from her own study of charter schools in New York City.

That study showed that charter schools in the city were having the opposite effect on their students’ achievement as the CREDO researchers found. Compared with their public school counterparts, Ms. Hoxby found, students in charter schools made more progress in closing achievement gaps with better-off peers in the suburbs. (“N.Y.C. Charters Found to Close Gaps,” Sept. 22, 2009.)

“She claims that we modeled charter schools in a particular way and that was inaccurate,” Margaret E. Raymond, CREDO’s director, said of Ms. Hoxby’s critique of the statistical analysis. “And her proof was based on her inaccurate specification.”

Ms. Raymond said Ms. Hoxby’s critique also failed to consider the many factors that the CREDO study took into account in its analysis.

Stakes High

“Imagine that we’re going to compare height and shoe size, and that we’re going to do it for all the people in the office,” Ms. Raymond said. “Those two things might actually correlate in a particular way because they’re only two factors. But if we’re going to predict shoe size based on many more factors, like age or parents’ height or parents’ shoe sizes, that’s a much more precise way of estimating.”

Criticism and Rebuttal

With charter schools a high-profile policy issue, researchers who came to opposite conclusions about their effect on student achievement are debating their analyses.

“She claims that we modeled charter schools in a particular way, and that was inaccurate. And her proof was based on her inaccurate speciication.”

Margaret E. Raymond
Director, Center for Research on Education Outcomes
Stanford University

“They assert that the proof proceeds in a way that it does not. Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

Caroline M. Hoxby
Professor of Economics
Stanford University

The CREDO researchers also argue in their memorandum that Ms. Hoxby’s estimates of the magnitude of the bias in their results was faulty.

The researchers’ arguments failed, however, to persuade Ms. Hoxby. In an e-mail response to Education Week, she said CREDO’s analysis contained “logically incorrect assertions.”

“They assert that the proof proceeds in a way that it does not,” Ms. Hoxby wrote. “Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

If the debate is taking on a bitter tone, it’s because the stakes are high. U.S. Secretary of Education Arne Duncan has been pressuring states to remove caps on charter schools in order to qualify for some of the $100 billion in economic-stimulus funds targeted to education. But opposition to the schools, which are public schools that operate largely free of the usual regulations, continues to come from teachers’ union officials and other education leaders.

“I don’t think the field of education research or policymakers are well served by scholars going back and forth with dueling memos, without peer review and without ample time to think it through,” said Sean F. Reardon, a colleague of both scholars at Stanford. “But I don’t think either side got it right,” he added.

An associate professor of education and sociology, Mr. Reardon is reviewing the New York City study. But he said he’s also not yet convinced that the CREDO study “does enough to ensure that it’s entirely free of bias.”

“Like any nonexperimental study,” he added, “it relies on some strong assumptions that are hard to justify.”

Part of the problem, he said, is that neither study provides enough statistical detail about the data and the methods used to help reviewers settle the debate.

Also, neither study has been published yet in a peer-reviewed journal. However, an earlier version of Ms. Hoxby’s report is posted as a working paper on the Web site of the National Bureau of Economic Research, which provides a forum for economists seeking feedback from the field on their work. Ms. Hoxby said her paper is also under consideration for publication by two journals.

The CREDO study was reviewed by four outside experts before its release this past June, according to a publicist. (“Study Casts Doubt on Charter School Results,” June 15, 2009)

The memo last week from CREDO suggests that it’s not surprising that the two studies should yield different results, given that a central finding of the CREDO study, which encompasses 15 states and the District of Columbia, is that charter school performance varies considerably from school to school.

“Some communities and states have gotten the policy right, and are able to demonstrate positive charter school results,” the CREDO researchers write. “The recent results for New York City indicate it, too, has a focused and effective charter policy.

“Our hope is that we can get beyond the distractions of this technical discussion,” the rebuttal adds, “and return to the larger policy issues.”

A version of this article appeared in the October 14, 2009 edition of Education Week as Charter School Researchers Spar Over Analyses

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Accelerate Learning with Project-Based Learning
Earlier this year, the George Lucas Educational Foundation released four new studies highlighting how project-based learning (PBL) helps accelerate student learning—across age groups, multiple disciplines, and different socio-economic statuses. With this year’s emphasis on unfinished
Content provided by SmartLab Learning
School & District Management Live Online Discussion Principal Overload: How to Manage Anxiety, Stress, and Tough Decisions
According to recent surveys, more than 40 percent of principals are considering leaving their jobs. With the pandemic, running a school building has become even more complicated, and principals' workloads continue to grow. If we
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Building Teacher Capacity for Social-Emotional Learning
Set goals that support adult well-being and social-emotional learning: register today!


Content provided by Panorama

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal California, Florida, and Other States Waiting on Green Light for Their COVID Relief Plans
The list of states with Ed. Dept. approval for their American Rescue Plan blueprints is growing steadily, but two big states aren't on it.
4 min read
Image shows an illustration of money providing relief against coronavirus.
DigitalVision Vectors/iStock/Getty
Federal What Is a School Shooting? Members of Congress Seek a Federal Definition, Reliable Data
A new bill would direct federal departments to track data related to school shootings, a term for which there is no federal definition.
Daniela Altimari, Hartford Courant
4 min read
Police respond to the scene of a shooting at Heritage High School in Newport News, Va., on Saturday Sept. 20, 2021. Newport News police Chief Steve Drew said two students were shot and taken to the hospital and neither injury was thought to be life-threatening. The chief said authorities believe the suspect and victims knew one another but did not provide details.
Police respond to the scene of a shooting at Heritage High School in Newport News, Va., on Saturday Sept. 20, 2021.
John C. Clark/AP Photo
Federal Is the Justice Dept. Silencing Parents or Stepping Up to Protect Educators?
Merrick Garland's move to use the FBI to help protect school officials from violence and harassment has drawn anger and praise.
5 min read
Sen. Josh Hawley, R-Mo., speaks during the Senate Judiciary Committee hearing to examine Texas's abortion law, Wednesday, Sept. 29, 2021 on Capitol Hill in Washington.
Sen. Josh Hawley, R-Mo., speaks during the Senate Judiciary Committee hearing to examine Texas's abortion law, Wednesday, Sept. 29, 2021 on Capitol Hill in Washington.
Tom Williams/Pool via AP
Federal Don't Use Federal COVID Aid to Undermine School Mask Rules, U.S. Treasury Tells Governor
Arizona Gov. Doug Ducey violated the intent of COVID aid programs by using them to discourage school mask mandates, an agency letter says.
2 min read
Arizona Gov. Doug Ducey speaks at the Arizona Capitol in Phoenix on Nov. 30, 2020. A program announced by Arizona's Republican governor last month to give private school vouchers to students whose parents object to school mask requirements has seen a surge of applications, with twice as many either completed or started than can be funded with the $10 million in federal coronavirus relief cash he earmarked for the program.
A program announced by Arizona's Republican Gov. Doug Ducey in September earmarks federal money to give private school vouchers to students whose parents object to public school mask requirements.
Ross D. Franklin/AP