Opinion
School Choice & Charters Opinion

What NAEP Can’t Tell Us About Charter School Effectiveness

By Jonathon Christensen & Lawrence Angel — December 06, 2005 5 min read
Since the data do not allow comparison of charter schools on more than one student attribute at a time, we cannot know whether the student populations of charter and conventional schools are very much alike or different.

Within a day of the release of the highly anticipated 2005 National Assessment of Educational Progress data in October, contending parties in the charter school debate had released claims that the findings supported their points of view. Meanwhile, we have continued to study the data to determine what conclusions can be drawn correctly. As it turns out, the claims from both sides overstate the extent to which determinations can be made about the effectiveness of the charter schools included in the sample.

The NAEP tests are part of an extensive effort undertaken by the National Center for Education Statistics to document the performance of America’s schools. NAEP data contain test results for a national sample of public and private school students in grades 4, 8, and 12; state-level samples include only public school students in these grades. The 2005 NAEP report draws on an impressive data set, containing math scores for 343,000 students and reading scores for 336,000 students from more than 17,600 schools nationwide.

The data are useful as an indication of how students are currently performing overall, both nationally and in each state. But they are much less useful as a tool for comparing the effectiveness of one type of school to that of another.

In 2003, NAEP began reporting separate results for charter schools. With the release of the 2005 NAEP data, charter school performance can be viewed at two points in time. Fourth graders in charter schools were first assessed in 2003, and a new set of charter 4th graders was included in the 2005 data.

The usefulness of these data for comparing charters over time is limited, however, since totally different groups of students were tested for the 2003 and 2005 reports. Moreover, the data are not helpful for comparing the effectiveness of charter schools with that of conventional public schools. Incomplete information on family background and the inability to track individual students over time allow school effectiveness to be confounded with student- background characteristics affecting performance. Further challenging reliability, the results are presented as broadly aggregated averages, which likely conceal considerable variation within groups.

It is very difficult to distinguish the impact of a school from other factors influencing the performance of a student. The quality of the school a student attends can affect his or her test scores, but so can other factors, such as parents’ education level, family background, and the quality of schools attended earlier.

— Susan Sanford

BRIC ARCHIVE

In a testing system like NAEP, where we know only a little bit about the family backgrounds of students and nothing about the schools they attended previously, it is hard to know what causes differences in schools’ test scores. If charter students perform at a higher or lower level than students in conventional public schools, the difference probably isn’t fully attributable to charter school attendance.

Assessing the relative effectiveness of charter and conventional public schools requires that researchers isolate the impact of school attendance from other influential factors. For 4th graders, comparisons of NAEP scores by group are strictly limited. Student-background factors, such as gender, race, family income, English-learner status, disability, or inner-city location, are controlled for only one at a time. But these factors are always found in combination, and charter schools and regular urban public schools serve many children who are educationally disadvantaged in multiple ways. Comparisons that look sensible might therefore be misleading.

It would be possible, for example, to compare charter and conventional public schools that serve the same proportions of African-American children. But the students in the two groups of schools could still be very different if, for example, charter school parents had much more (or much less) education than parents in the conventional public schools. Since NAEP data do not allow comparison of charter schools on more than one student attribute at a time, we cannot know whether the student populations of charter and conventional schools are very much alike or different.

NAEP data also make it impossible to distinguish different kinds of charter schools. This is important, because charter schools differ significantly from state to state, and even within particular localities. Reporting on charter schools’ performance as a whole conceals valuable information about the impact of various curricula, organizational structures, and accountability systems.

Perhaps the biggest potential pitfall of interpreting NAEP data is making an inference about an individual or school based on aggregate data for a group.

In examining the aggregate data for charter schools in Delaware, for example, one might discover that the average scale score for 4th graders tested in mathematics is 235. Of course, this is an average, and one cannot simply conclude that most 4th graders in Delaware score around 235. Maybe all Delaware charter school students have about the same score. Maybe student scores range from under 100 to over 500, with big differences between ethnic groups or between city and rural schools. The data available don’t let us determine which is true.

Presenting results on average, as NAEP is designed to do, conceals important variations among individual students and schools. At the same time, it might wrongly give the impression of causal relationships where none exist. For example, a reasonable interpretation of the recent NAEP results might be that “4th grade charter school students, on average, are earning slightly better NAEP mathematics scores than the charter students included in the test two years ago.” But the students and the schools included in this year’s NAEP are not the same as the ones included two years ago. So it is impossible to say that charter schools are teaching their students better than before, or that a definable group of students is learning more than in the past.

No matter how hard analysts try, it is not possible to squeeze good results on charter school effectiveness out of the NAEP data. There are many serious studies under way and about to be published, and though these too will be controversial, they can do a much better job of comparing like with like and separating school results from the effects of student characteristics.

Related Tags:

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Culturally Relevant Pedagogy to Advance Educational Equity
Schools are welcoming students back into buildings for full-time in-person instruction in a few short weeks and now is the perfect time to take a hard look at both our practices and systems to build
Content provided by PowerMyLearning
Classroom Technology Webinar Making Big Technology Decisions: Advice for District Leaders, Principals, and Teachers
Educators at all levels make decisions that can have a huge impact on students. That’s especially true when it comes to the use of technology, which was activated like never before to help students learn
Professional Development Webinar Expand Digital Learning by Expanding Teacher Training
This discussion will examine how things have changed and offer guidance on smart, cost-effective ways to expand digital learning efforts and train teachers to maximize the use of new technologies for learning.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School Choice & Charters Virtual Charters in Hot Water Again. Accusations of Fraud Prompt $150M Lawsuit
Indiana officials seek to recoup more than $150 million they say was either wrongly obtained or misspent by a consortium of virtual schools.
Arika Herron, The Indianapolis Star
2 min read
Indiana's attorney general Todd Rokita speaks at a news conference on Sept. 16, 2020, in Indianapolis. Rokita filed a lawsuit against a group of online charter schools accused of defrauding the state out of millions of dollars Thursday, July 8, 2021.
Indiana's attorney general Todd Rokita speaks at a news conference on Sept. 16, 2020, in Indianapolis.
Darron Cummings/AP
School Choice & Charters How the Pandemic Helped Fuel the Private School Choice Movement
State lawmakers got a new talking point as they pushed to create and expand programs to send students to private schools.
8 min read
Collage showing two boys in classroom during pandemic wearing masks with cropped photo of feet and arrows going in different directions.
Collage by Gina Tomko/EducationWeek (Images: Getty)
School Choice & Charters Opinion Taking Stock After 30 Years of Charter Schools
Rick Hess speaks with Nina Rees, president and CEO of the National Alliance for Public Charter Schools, on charter schools turning 30.
8 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School Choice & Charters In Fight Over Millions of Dollars for Charter Schools, a Marijuana Tax May Bring Peace
The Oklahoma State Board of Education voted unanimously to rescind a polarizing lawsuit settlement, pending certain stipulations.
Nuria Martinez-Keel, The Oklahoman
3 min read
Money bills cash funds close up Getty
Getty