Corrected: This article presented data on student test scores incorrectly. The study found that 4th grade students in a nationally representative sample of charter schools scored 4.2 points behind students in regular public schools on the National Assessment of Educational Progress in reading, and 4.7 points behind regular public-school students in mathematics, when various student characteristics were considered. Those score differences were measured on the NAEP achievement scale of zero to 500.
In discussing a new federal study on charter schools last week, the commissioner of statistics for the U.S. Department of Education reiterated that his office should not be initiating analyses such as that one and a recent comparison of public and private schools, both of which he believes rely too much on subjective judgments.
The two studies—which used data from the National Assessment of Educational Progress that were adjusted for student and school differences—reported generally favorable academic results for regular public schools when compared with private schools and public charter schools. The studies, though, did not track scores over time, and could not take into account students’ prior achievement.
Mark S. Schneider, the commissioner of the National Center for Education Statistics, said of the private school study earlier this month that it was not proper for the NCES to have directed a report that went so far in making judgments about how to interpret raw school data.
He repeated such concerns during a conference call last week on the charter study, saying that both analyses, which were initiated under his predecessor, could lead outsiders to question the statistics center’s impartiality.
“We do not have plans to replicate this analysis using the 2005 or eventually the 2007 data,” Mr. Schneider said of the charter study during the Aug. 22 call. “That task is best left to individual researchers, who should be competing with each other in the marketplace of ideas.”
Read the related story,
The charter study, after taking into account a range of student and school characteristics such as race, family income, and other socioeconomic variables, found that charter schools on average trailed regular public schools in 4th grade achievement in both reading and mathematics. The study found the average reading score for the 150 charters sampled was 4.2 percentage points behind that for a much larger sample of regular public schools. For mathematics, charter schools were 4.7 percentage points behind. (Reanalysis of NAEP Scores Finds Charter Schools Lagging, this issue.)
The commissioner initially had voiced reservations about the NCES’ involvement in the private school study at an Aug. 4 meeting of the National Assessment Governing Board in McLean, Va., and elaborated on those comments in an interview afterward. The governing board sets policy for the National Assessment of Educational Progress, or NAEP, which the NCES administers.
The report on public and private schools, which received wide attention when it was released last month, showed public school students generally performing as well as or better than their private school peers in reading and math, when researchers adjusted for race, socioeconomic status, and other characteristics. One exception was 8th grade reading, in which private school children did better than their public school peers. (“Public Schools Fare Well Against Private Schools in Study,” July 26, 2006.)
“This is not what we should be doing,” Mr. Schneider said of the private school study in an interview with Education Week. “It’s one thing for [an academic researcher] to put out a study. That’s a totally different story than if NCES, as a statistical agency, is weighing in on that front.”
The NCES is one of three research centers housed within the Institute of Education Sciences, the Education Department’s research arm as reorganized by Congress in 2002. Although the institute is a part of the department, it is supposed to conduct its work free of political influence.
The IES director can prepare and publish any research without the approval of the education secretary. While the heads of two of the institute’s research centers are appointed by the IES director, the NCES commissioner is appointed by the president. Mr. Schneider had no say in the NCES’ decision to conduct the private school and charter school studies. Those projects were begun under Robert Lerner, Mr. Schneider’s predecessor. Mr. Lerner could not be reached for comment.
Like Mr. Lerner, Mr. Schneider was appointed by President Bush; his term expires in June 2009.
A Departure for NCES
Henry I. Braun, a senior educational researcher for the Educational Testing Service, a nonprofit research and testing organization in Princeton, N.J., led both studies for the federal statistical agency.
Each of them relied on a technique known as hierarchical linear modeling, or HLM, which allows scores to be compared while simultaneously taking into consideration multiple characteristics of students and schools. This method allowed for a more detailed analysis than was used during a 2004 NCES comparison of charter and regular public schools.
But using the HLM technique requires researchers to choose which variables to consider—such as race, gender, and various measures of socioeconomic status—as the ETS did on the study of public and private schools and the charter study. Although it is common for outside researchers to crunch federal data using the HLM method, the NCES has typically not engaged in such work, Mr. Braun and others said.
“I think it was clear from the beginning that this was something of a departure for NCES,” said Mr. Braun of his private school study.
Peggy Carr, the statistical agency’s associate commissioner, said the NCES had made only “very limited” use of the HLM technique in previous studies.
The ETS conducted the private and charter school studies under an existing, five-year contract with the NCES, Ms. Carr said. Both studies underwent peer review by outside experts, she said. Each study cost about $130,000.
While the NCES publishes such reports and provides general guidance to their authors—such as the decision to use the HLM method—the research itself was conducted entirely by the ETS, and decisions about what variables to consider and other procedures were left to the contractor, Ms. Carr said.
IES Chief Concurs
Mr. Schneider emphasized that he was not questioning the merits of the NCES reports, which he described as high-quality efforts. But the commissioner said that in the future he would not support having the NCES produce a report in which researchers made similarly subjective judgments about which student characteristics to consider.
“Our job is to collect the data and get it out the door,” Mr. Schneider said in an interview after the NAEP governing board meeting earlier this month. The problem, he said, is that in studies like the private school report, “you’re always having to make choices about variables.”
After the publication of the study on public and private schools, Paul E. Peterson, a professor of government at Harvard University, analyzed the data using different variables and found that private school students’ performance topped that of public school students. (“Data Reanalysis Finds Test-Score Edge for Private Schools,” Aug. 9, 2006.)
The head of the Institute of Education Sciences echoed Mr. Schneider’s opinion. In an e-mail, IES Director Grover J. “Russ” Whitehurst said studies using hierarchical linear modeling and similar methods of analysis are “not a priority” for the NCES, because they take time away from other projects, and private-sector researchers have the ability to conduct the same work. Mr. Whitehurst also noted that such interpretive reports almost always produce debates about the methodology used.
“There are typically several legitimate ways to carry out complex data analyses,” Mr. Whitehurst wrote in the e-mail. “Experts will differ on which approach is best and how the results should be interpreted. … Advocacy groups that don’t like the results will frequently assert that the analytic approach was biased. It is best for NCES, as a federal statistical agency, to stay out of that fray.”
Sensitivity Cited
Gerald E. Sroufe, the director of government relations for the American Educational Research Association, a 25,000-member professional association based in Washington, essentially agreed.
“It should provide the raw material,” Mr. Sroufe said of the NCES. Having the statistics center produce studies using HLM and similar methods “is probably not a good idea, particularly involving work that is so politically sensitive,” he said.
The study on public and private schools came out the week before Secretary of Education Margaret Spellings and Republican lawmakers unveiled a $100 million proposal to make vouchers for private school tuition available under the No Child Left Behind Act. In questioning the wisdom of that proposal, critics cited the NCES report showing public school students outperforming their private school peers.
Despite that debate, Mr. Schneider said that his concerns were his own, and that Bush administration officials did not try to influence his views on the subject. “The NCES,” he said, “is quite insulated from that kind of political pressure.”
