Federal

Scholars Spar Over Research Methods Used to Evaluate Charters

By Debra Viadero — October 08, 2009 4 min read

The authors of a recent national study that found students in regular public schools outperforming their charter school peers are rebutting criticism that their research suffered from a “serious mathematical mistake” that negatively skewed the results.

In a memorandum posted online last week, researchers from the Center for Research on Education Outcomes, or CREDO, at Stanford University responded to criticism made last month by their Stanford colleague Caroline M. Hoxby. Ms. Hoxby, an economics professor, issued a memo critiquing the CREDO study in tandem with results from her own study of charter schools in New York City.

That study showed that charter schools in the city were having the opposite effect on their students’ achievement as the CREDO researchers found. Compared with their public school counterparts, Ms. Hoxby found, students in charter schools made more progress in closing achievement gaps with better-off peers in the suburbs. (“N.Y.C. Charters Found to Close Gaps,” Sept. 22, 2009.)

“She claims that we modeled charter schools in a particular way and that was inaccurate,” Margaret E. Raymond, CREDO’s director, said of Ms. Hoxby’s critique of the statistical analysis. “And her proof was based on her inaccurate specification.”

Ms. Raymond said Ms. Hoxby’s critique also failed to consider the many factors that the CREDO study took into account in its analysis.

Stakes High

“Imagine that we’re going to compare height and shoe size, and that we’re going to do it for all the people in the office,” Ms. Raymond said. “Those two things might actually correlate in a particular way because they’re only two factors. But if we’re going to predict shoe size based on many more factors, like age or parents’ height or parents’ shoe sizes, that’s a much more precise way of estimating.”

Criticism and Rebuttal

With charter schools a high-profile policy issue, researchers who came to opposite conclusions about their effect on student achievement are debating their analyses.

“She claims that we modeled charter schools in a particular way, and that was inaccurate. And her proof was based on her inaccurate speciication.”

Margaret E. Raymond
Director, Center for Research on Education Outcomes
Stanford University

“They assert that the proof proceeds in a way that it does not. Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

Caroline M. Hoxby
Professor of Economics
Stanford University

The CREDO researchers also argue in their memorandum that Ms. Hoxby’s estimates of the magnitude of the bias in their results was faulty.

The researchers’ arguments failed, however, to persuade Ms. Hoxby. In an e-mail response to Education Week, she said CREDO’s analysis contained “logically incorrect assertions.”

“They assert that the proof proceeds in a way that it does not,” Ms. Hoxby wrote. “Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

If the debate is taking on a bitter tone, it’s because the stakes are high. U.S. Secretary of Education Arne Duncan has been pressuring states to remove caps on charter schools in order to qualify for some of the $100 billion in economic-stimulus funds targeted to education. But opposition to the schools, which are public schools that operate largely free of the usual regulations, continues to come from teachers’ union officials and other education leaders.

“I don’t think the field of education research or policymakers are well served by scholars going back and forth with dueling memos, without peer review and without ample time to think it through,” said Sean F. Reardon, a colleague of both scholars at Stanford. “But I don’t think either side got it right,” he added.

An associate professor of education and sociology, Mr. Reardon is reviewing the New York City study. But he said he’s also not yet convinced that the CREDO study “does enough to ensure that it’s entirely free of bias.”

“Like any nonexperimental study,” he added, “it relies on some strong assumptions that are hard to justify.”

Part of the problem, he said, is that neither study provides enough statistical detail about the data and the methods used to help reviewers settle the debate.

Also, neither study has been published yet in a peer-reviewed journal. However, an earlier version of Ms. Hoxby’s report is posted as a working paper on the Web site of the National Bureau of Economic Research, which provides a forum for economists seeking feedback from the field on their work. Ms. Hoxby said her paper is also under consideration for publication by two journals.

The CREDO study was reviewed by four outside experts before its release this past June, according to a publicist. (“Study Casts Doubt on Charter School Results,” June 15, 2009)

The memo last week from CREDO suggests that it’s not surprising that the two studies should yield different results, given that a central finding of the CREDO study, which encompasses 15 states and the District of Columbia, is that charter school performance varies considerably from school to school.

“Some communities and states have gotten the policy right, and are able to demonstrate positive charter school results,” the CREDO researchers write. “The recent results for New York City indicate it, too, has a focused and effective charter policy.

“Our hope is that we can get beyond the distractions of this technical discussion,” the rebuttal adds, “and return to the larger policy issues.”

A version of this article appeared in the October 14, 2009 edition of Education Week as Charter School Researchers Spar Over Analyses

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Professional Development Webinar
Building Leadership Excellence Through Instructional Coaching
Join this webinar for a discussion on instructional coaching and ways you can link your implement or build on your program.
Content provided by Whetstone Education/SchoolMint
Teaching Webinar Tips for Better Hybrid Learning: Ask the Experts What Works
Register and ask your questions about hybrid learning to our expert panel.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Families & the Community Webinar
Family Engagement for Student Success With Dr. Karen Mapp
Register for this free webinar to learn how to empower and engage families for student success featuring Karen L. Mapp.
Content provided by Panorama Education & PowerMyLearning

EdWeek Top School Jobs

DevOps Engineer
Portland, OR, US
Northwest Evaluation Association
User Experience Analyst
Portland, OR, US
Northwest Evaluation Association
Senior Business Analyst - 12 Month Contract
Portland, OR, US
Northwest Evaluation Association
Senior Director Marketing
Philadelphia, Pennsylvania, United States
Camelot Education

Read Next

Federal Biden Announces Goal to Get Educators the COVID-19 Vaccine This Month
President Joe Biden pushes states to get educators at least one dose by the end of March to help schools resume in-person learning.
4 min read
John Battle High School teacher Jennifer Daniel receives her COVID-19 vaccine on Jan. 11, 2021. Teachers received their first vaccine during an all-day event at the Virginia Highlands Higher Education Center in Abingdon, Va.
John Battle High School teacher Jennifer Daniel receives her COVID-19 vaccine on Jan. 11, 2021. Teachers received their first vaccine during an all-day event at the Virginia Highlands Higher Education Center in Abingdon, Va.
David Crigger/Bristol Herald Courier via AP
Federal Explainer Miguel Cardona, U.S. Secretary of Education: Background and Achievements
Background and highlights of Miguel Cardona's tenure as the twelfth U.S. Secretary of Education.
Education Week Library
2 min read
Miguel Cardona, President-elect Joe Biden's nominee for Secretary of Education, speaks after being introduced at The Queen Theater in Wilmington, Del., on Dec. 23, 2020.
Miguel Cardona, U.S. Secretary of Education, speaks after being put forward for the position by then-President-elect Joe Biden in December 2020.
Carolyn Kaster/AP
Federal Senate Confirms Miguel Cardona as Education Secretary
The former Connecticut education commissioner got his start as an elementary school teacher and was a principal and school administrator.
2 min read
Miguel Cardona, President-elect Joe Biden's nominee for Secretary of Education, speaks after being introduced at The Queen Theater in Wilmington, Del., Wednesday, Dec. 23, 2020.
Miguel Cardona was confirmed by the Senate to serve as U.S. Secretary of Education. The former Connecticut education commissioner has worked as a teacher, principal, and district administrator.
Carolyn Kaster/AP
Federal Biden Legal Team Steps Back From Trump Stance on Transgender Female Sports Participation
The Education Department's office for civil rights pulls a letter that said Connecticut's transgender-inclusive policy violates Title IX.
4 min read
Bloomfield High School transgender athlete Terry Miller, second from left, wins the final of the 55-meter dash over transgender athlete Andraya Yearwood, far left, and other runners in the Connecticut girls Class S indoor track meet at Hillhouse High School in New Haven, Conn on Feb. 7, 2019. Transgender athletes are getting an ally in the White House next week as they seek to participate as their identified gender in high school and college sports. Attorneys on both sides say they expect President-elect Joe Biden’s Department of Education will switch sides in legal battles that could go a long way in determining whether transgender athletes are treated by the sex on their birth certificates or by how they identify.
Bloomfield High School transgender athlete Terry Miller, second from left, wins over transgender athlete Andraya Yearwood, far left, and other runners in an event in New Haven, Conn. The two transgender athletes are at the center of a legal fight in Connecticut over the participation of transgender female athletes in girls' or women's sports.
Pat Eaton-Robb/AP