Harvard University researchers publicized findings last week calling into question the methodology of recent studies finding that students at public schools did as well as or better than their private school peers on some standardized tests when scores were adjusted for certain student characteristics.
Paul E. Peterson, a professor at Harvard’s John F. Kennedy School of Government, found that when he and graduate student Elena Llaudet reanalyzed data from the National Assessment of Educational Progress using different variables to adjust for student characteristics, students at private schools came out on top of those in public schools in almost all areas.
That conclusion was nearly the opposite of a study recently released by the U.S. Department of Education, as well as an earlier study by two University of Illinois professors. (“Public Schools Fare Well Against Private Schools in Study,” July 26, 2006.)
“On the Public-Private School Achievement Debate” is available from the Program on Education Policy and Governance at Harvard University.
In all three studies, researchers adjusted for characteristics such as race and socioeconomic status, but they based the adjustment on different information that had been reported to NAEP.
Mr. Peterson said that none of the three studies can conclude with any confidence that one group of schools does better than the other, because the NAEP data provide only a snapshot of how students did on tests at one point in time, rather than what they learned over a period of time.
“We aren’t offering this study as definitive evidence,” Mr. Peterson said. “We’re offering it as strong evidence that the methods used by the other two studies are defective.”
Henry Braun, the senior author of the NCES report and a senior educational researcher for the Educational Testing Service in Princeton, N.J., acknowledged that Mr. Peterson has raised some important issues regarding the variables used in the NCES study. But he said the variables used by Mr. Peterson are equally problematic.
“Because of the variables he’s using, it may be that he is underadjusting for disadvantage in the public school sector,” he said.
Christopher Lubienski and Sarah Theule Lubienski, a husband-and-wife research team at the University of Illinois at Urbana-Champaign who published a study in January that Mr. Peterson is revisiting, argued in an interview last week that the variables chosen by Mr. Peterson are flawed and inferior to the ones they used.
Different Classifications?
The Harvard team relied largely on information about student characteristics reported by the students themselves, rather than information reported by public and private school administrators. Mr. Peterson contends that in comparisons of public and private schools, data reported by administrators based on their schools’ participation in federal programs, such as the federal subsidized lunch program, is not reliable because both kinds of schools have very different involvement in those programs and classify their students in different ways.
Ms. Lubienski acknowledged that classification differences between public and private schools pose a problem. But she argued that the Harvard team is “throwing the baby out with the bath water” to exclude data such as whether students are identified as having limited proficiency in English or have individualized education programs when controlling for student background.
She said some of the variables Mr. Peterson accounts for also have flaws. He controls for the education level of students’ parents, for example, which Ms. Lubienski sees as a problem because some 4th graders who reported that information to NAEP likely don’t know their parents’ education levels.
Mr. Braun added that Mr. Peterson’s use of parental education to adjust for socioeconomic level is flawed because he didn’t account for such nuances as whether both parents or only one has a college education.
The federal study, released July 14 by the National Center for Education Statistics, found that when data are adjusted for student characteristics, 4th and 8th grade public school students perform as well as or better than private school students in reading and math, with the exception of 8th grade reading, where children in private schools do better than their public school peers.
Those results were similar to those found by the Lubienskis, though they looked only at NAEP math scores.
