Assessment

NAEP Gap Continuing for Charters

Sector’s Scores Lag in Three Out of Four Main Categories
May 19, 2008 8 min read
  • Save to favorites
  • Print

Includes updates and/or revisions.

Nearly four years after a front-page story in The New York Times sparked a fierce debate by suggesting that charter school students nationally were lagging academically behind their peers in regular public schools, the national testing program that informed the controversy has generated far more data for researchers and advocates to scrutinize.

Yet the more recent findings from the National Assessment of Educational Progress have garnered much less attention and analysis than the 2003 results.

The picture that emerges from the growing data set appears mixed for charter schools. While many analysts urge caution in using NAEP to judge the 4,300-school charter sector, the latest data do not bolster the early hopes of charter advocates that the sector as a whole would significantly outperform regular public schools.

The overall scores of charter students tested in 2007 in the nationally representative assessment program were lower than for students in regular public schools in 4th grade reading and mathematics, and in 8th grade math, all by statistically significant margins.

In 4th grade reading, charter students had an average score of 214, compared with 220 for regular public schools, on a 500-point scale. Looked at another way, 59 percent of charter students scored at or above the “basic” reading level on the NAEP test, compared with 66 percent in other public schools.

But in 8th grade reading, charter students appeared to essentially close a gap from 2005, with charter and regular public school students scoring about the same in 2007.

Digging deeper in the data reveals a more complex story, though limited sample sizes for charter schools make many score differences hard to interpret with confidence. For low-income black students—a key population served by many urban charters—the 2007 performance gaps between charter and noncharter schools generally appeared smaller than those between the two sectors’ populations as a whole, and none was large enough to be deemed statistically significant.

Hispanic charter students, meanwhile, appeared to do about as well as or better than their peers in regular public schools across grades and subjects. But here, too, limited sample sizes make the differences too small to state with confidence.

Piece of the Puzzle

Researchers emphasize that because of NAEP’s design, the program has serious limitations in assessing charter schools, or comparing them with other public schools. Some experts argue that the tests are altogether ill-suited for the purpose.

The congressionally mandated NAEP—known as “the nation’s report card”—provides a snapshot of performance at a single point in time, testing different students each cycle. It also does not take into account prior achievement.

Even with those and other caveats, some researchers—though not all—suggest the results are a useful part of the growing research base on charter achievement.

“I personally don’t believe we’re going to have a single, definitive study to tell us whether charters are working,” said Gary J. Miron, a principal research associate at the evaluation center at Western Michigan University, who studies charter schools. “The NAEP is one more piece of the puzzle. … And now we have multiple snapshots.”

The debate over charter schools and NAEP first came to public prominence following an August 2004 story in The New York Times. The article focused on an analysis conducted by the American Federation of Teachers, which had called for a national moratorium on new charter schools in 2002.

The AFT study used reading and math scores in 2003, the first time charter students’ NAEP scores were reported separately. It concluded that charter students trailed those in regular public schools as a whole, and also when results were broken down by such subgroups as black students or students eligible for free or reduced-price lunches.

The report and news story spurred an outcry from charter advocates and some researchers, who suggested its methodology was flawed and the results misleading. The 2003 NAEP data were the subject of multiple re-evaluations, including a federal analysis issued in 2006 finding that scores in charter schools, taking into account a range of background characteristics of students and schools, trailed those in regular public schools that year in reading and math. (“Reanalysis of NAEP Scores Finds Charter Schools Lagging,” Aug. 30, 2006.)

Adding to the Debate?

Charter schools, which are publicly funded but largely independent, have been the focus of a growing body of studies. Many experts say the overall findings so far are inconclusive, with a blend of positive, neutral, and negative outcomes for charter schools compared with other public schools.

Todd M. Ziebarth, a senior policy analyst at the National Alliance for Public Charter Schools, a Washington research and advocacy group, said his group is keenly aware of NAEP’s limitations, but keeps an eye on the results. “They’re important to look at and learn from, but you learn so little compared to [other] assessments,” he said.

Chester E. Finn Jr., the president of the Thomas B. Fordham Foundation, a Washington think tank that is the authorizer for a set of Ohio charter schools, was among the charter advocates who urged the separate reporting of results for charter students on the 2003 NAEP tests. He said he believed it would be good for charter schools and policymakers.

“I hoped they also would be good for traditional public schools, and might show them up,” he said.

Mr. Finn said he now believes the most important distinctions are between different kinds of charters, such as those that operate in networks, or from state to state where different governing laws apply.

“I’m not very interested in the average performance of charters,” he said. “The word ‘charter’ signals so little about them, and the diversity within that universe is at least as great as the diversity outside it.”

The Charter Sample

The NAEP sample of charter students is designed to be nationally representative, but is dwarfed by the number of students tested in regular public schools.

Recent samples have ranged from 2,300 to 3,300 charter students for each grade level in reading, and from 150 to more than 200 schools. By contrast, in 2007, the national public school sample in 4th grade reading included 183,400 students from 7,310 schools, with somewhat lower 8th grade figures.

The population that charter schools typically serve is substantially different from that served by public schools as a whole, complicating comparisons across school types. In 2007, the NAEP charter sample had more than twice as many black students on a percentage basis and far more students living in cities—groups that generally score below national averages—than did the pool of NAEP test-takers overall.

And the charter population itself is changing, the test samples suggest. For example, 53 percent of charter students tested in 4th grade math in 2007 were eligible for free or reduced-price lunches, up from 42 percent in 2003.

With three rounds of reading and math scores for 4th grade charter students, the NAEP results have not shown clear trends. In reading, charter students’ scores apparently rose in 2005 but dropped slightly in 2007; the changes were not large enough, however, to be statistically significant. With 4th graders in regular public schools posting reading gains in 2007, the charter students slipped behind them.

In 4th grade math, charter students have shown what appear to be steady gains from 2003 to 2007—again, not statistically significant—but made little headway in closing the gap with students in regular public schools, who saw a similar growth trend.

Looking at students by race and income status, as measured by eligibility for free or reduced-price lunches, seems to alter the picture.

In 4th grade math, charter students overall were 5 points behind noncharter students in both 2005 and 2007 on the 500-point scale, statistically significant differences.

But for low-income black students, the difference between those in charter schools and those in regular public schools was 1 point in 2005 and 2 points in 2007, neither of which was statistically significant. Measured another way, 56 percent of low-income black students from charter schools scored at the basic level or above in 2007, compared with 59 percent of such students in regular public schools.

“Certainly, the raw data suggest charter students are behind,” said Sarah Theule Lubienski, an associate professor of education at the University of Illinois at Urbana-Champaign, after reviewing the recent results for 4th grade students in math and reading. But she said that when controlling for race and low-income status, it appears that “neither one is ahead.”

In a press release last September, the national charter alliance highlighted the gains of 8th grade charter students in 2007, though those apparent increases were not deemed statistically significant. The average scale score rose from 255 in 2005 to 260 in 2007 in reading, and from 268 to 273 in math.

Growth Rates Differ?

The alliance concluded that charter students’ achievement “increased at a notably faster rate” than that of their peers in regular public schools. It also noted stronger 8th grade gains in certain categories, such as the reading scores for African-American and low-income students.

The charter alliance also noted the performance of Hispanic charter students. In 8th grade math, their average score was 9 points higher than for Hispanics in regular public schools in 2005, and 8 points higher in 2007. Because of the limited charter sample size, though, the differences were not deemed statistically significant.

For its part, the AFT has been quiet about the recent NAEP data. Asked about the results, union spokesman Dan Murphy said: “Basically, what the NAEP data suggests is what we already know. There are some excellent charter schools and some charter schools that fall short. … And the same goes for regular public schools.”

Coverage of new schooling arrangements and classroom improvement efforts is supported by a grant from the Annenberg Foundation.
A version of this article appeared in the May 21, 2008 edition of Education Week as NAEP Gap Continuing for Charters

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Whitepaper
Design for Improvement: The Case for a New Accountability System
Assessments in more frequent intervals provide useful feedback on what students actually study. New curriculum-aligned assessments can le...
Content provided by Cognia
Assessment The 5 Burning Questions for Districts on Grading Reforms
As districts rethink grading policies, they consider the purpose of grades and how to make them more reliable measures of learning.
5 min read
Grading reform lead art
Illustration by Laura Baker/Education Week with E+ and iStock/Getty
Assessment As They Revamp Grading, Districts Try to Improve Consistency, Prevent Inflation
Districts have embraced bold changes to make grading systems more consistent, but some say they've inflated grades and sent mixed signals.
10 min read
Close crop of a teacher's hands grading a stack of papers with a red marker.
E+
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Sponsor
Fewer, Better Assessments: Rethinking Assessments and Reducing Data Fatigue
Imagine a classroom where data isn't just a report card, but a map leading students to their full potential. That's the kind of learning experience we envision at ANet, alongside educators
Content provided by Achievement Network
Superintendent Dr. Kelly Aramaki - Watch how ANet helps educators
Photo provided by Achievement Network