School Choice & Charters

NCES Calls for Sticking to the Stats

By Sean Cavanagh & Erik W. Robelen — August 29, 2006 | Corrected: February 22, 2019 7 min read
  • Save to favorites
  • Print

Corrected: This article presented data on student test scores incorrectly. The study found that 4th grade students in a nationally representative sample of charter schools scored 4.2 points behind students in regular public schools on the National Assessment of Educational Progress in reading, and 4.7 points behind regular public-school students in mathematics, when various student characteristics were considered. Those score differences were measured on the NAEP achievement scale of zero to 500.

In discussing a new federal study on charter schools last week, the commissioner of statistics for the U.S. Department of Education reiterated that his office should not be initiating analyses such as that one and a recent comparison of public and private schools, both of which he believes rely too much on subjective judgments.

The two studies—which used data from the National Assessment of Educational Progress that were adjusted for student and school differences—reported generally favorable academic results for regular public schools when compared with private schools and public charter schools. The studies, though, did not track scores over time, and could not take into account students’ prior achievement.

Mark S. Schneider

Mark S. Schneider, the commissioner of the National Center for Education Statistics, said of the private school study earlier this month that it was not proper for the NCES to have directed a report that went so far in making judgments about how to interpret raw school data.

He repeated such concerns during a conference call last week on the charter study, saying that both analyses, which were initiated under his predecessor, could lead outsiders to question the statistics center’s impartiality.

“We do not have plans to replicate this analysis using the 2005 or eventually the 2007 data,” Mr. Schneider said of the charter study during the Aug. 22 call. “That task is best left to individual researchers, who should be competing with each other in the marketplace of ideas.”

The charter study, after taking into account a range of student and school characteristics such as race, family income, and other socioeconomic variables, found that charter schools on average trailed regular public schools in 4th grade achievement in both reading and mathematics. The study found the average reading score for the 150 charters sampled was 4.2 percentage points behind that for a much larger sample of regular public schools. For mathematics, charter schools were 4.7 percentage points behind. (Reanalysis of NAEP Scores Finds Charter Schools Lagging, this issue.)

The commissioner initially had voiced reservations about the NCES’ involvement in the private school study at an Aug. 4 meeting of the National Assessment Governing Board in McLean, Va., and elaborated on those comments in an interviewafterward. The governing board sets policy for the National Assessment of Educational Progress, or NAEP, which the NCES administers.

The report on public and private schools, which received wide attention when it was released last month, showed public school students generally performing as well as or better than their private school peers in reading and math, when researchers adjusted for race, socioeconomic status, and other characteristics. One exception was 8th grade reading, in which private school children did better than their public school peers. (“Public Schools Fare Well Against Private Schools in Study,” July 26, 2006.)

“This is not what we should be doing,” Mr. Schneider said of the private school study in an interview with Education Week. “It’s one thing for [an academic researcher] to put out a study. That’s a totally different story than if NCES, as a statistical agency, is weighing in on that front.”

The NCES is one of three research centers housed within the Institute of Education Sciences, the Education Department’s research arm as reorganized by Congress in 2002. Although the institute is a part of the department, it is supposed to conduct its work free of political influence.

The IES director can prepare and publish any research without the approval of the education secretary. While the heads of two of the institute’s research centers are appointed by the IES director, the NCES commissioner is appointed by the president. Mr. Schneider had no say in the NCES’ decision to conduct the private school and charter school studies. Those projects were begun under Robert Lerner, Mr. Schneider’s predecessor. Mr. Lerner could not be reached for comment.

Like Mr. Lerner, Mr. Schneider was appointed by President Bush; his term expires in June 2009.

A Departure for NCES

Henry I. Braun, a senior educational researcher for the Educational Testing Service, a nonprofit research and testing organization in Princeton, N.J., led both studies for the federal statistical agency.

Each of them relied on a technique known as hierarchical linear modeling, or HLM, which allows scores to be compared while simultaneously taking into consideration multiple characteristics of students and schools. This method allowed for a more detailed analysis than was used during a 2004 NCES comparison of charter and regular public schools.

But using the HLM technique requires researchers to choose which variables to consider—such as race, gender, and various measures of socioeconomic status—as the ETS did on the study of public and private schools and the charter study. Although it is common for outside researchers to crunch federal data using the HLM method, the NCES has typically not engaged in such work, Mr. Braun and others said.

“I think it was clear from the beginning that this was something of a departure for NCES,” said Mr. Braun of his private school study.

Peggy Carr, the statistical agency’s associate commissioner, said the NCES had made only “very limited” use of the HLM technique in previous studies.

The ETS conducted the private and charter school studies under an existing, five-year contract with the NCES, Ms. Carr said. Both studies underwent peer review by outside experts, she said. Each study cost about $130,000.

While the NCES publishes such reports and provides general guidance to their authors—such as the decision to use the HLM method—the research itself was conducted entirely by the ETS, and decisions about what variables to consider and other procedures were left to the contractor, Ms. Carr said.

IES Chief Concurs

Mr. Schneider emphasized that he was not questioning the merits of the NCES reports, which he described as high-quality efforts. But the commissioner said that in the future he would not support having the NCES produce a report in which researchers made similarly subjective judgments about which student characteristics to consider.

“Our job is to collect the data and get it out the door,” Mr. Schneider said in an interview after the NAEP governing board meeting earlier this month. The problem, he said, is that in studies like the private school report, “you’re always having to make choices about variables.”

After the publication of the study on public and private schools, Paul E. Peterson, a professor of government at Harvard University, analyzed the data using different variables and found that private school students’ performance topped that of public school students. (“Data Reanalysis Finds Test-Score Edge for Private Schools,” Aug. 9, 2006.)

The head of the Institute of Education Sciences echoed Mr. Schneider’s opinion. In an e-mail, IES Director Grover J. “Russ” Whitehurst said studies using hierarchical linear modeling and similar methods of analysis are “not a priority” for the NCES, because they take time away from other projects, and private-sector researchers have the ability to conduct the same work. Mr. Whitehurst also noted that such interpretive reports almost always produce debates about the methodology used.

“There are typically several legitimate ways to carry out complex data analyses,” Mr. Whitehurst wrote in the e-mail. “Experts will differ on which approach is best and how the results should be interpreted. … Advocacy groups that don’t like the results will frequently assert that the analytic approach was biased. It is best for NCES, as a federal statistical agency, to stay out of that fray.”

Sensitivity Cited

Gerald E. Sroufe, the director of government relations for the American Educational Research Association, a 25,000-member professional association based in Washington, essentially agreed.

“It should provide the raw material,” Mr. Sroufe said of the NCES. Having the statistics center produce studies using HLM and similar methods “is probably not a good idea, particularly involving work that is so politically sensitive,” he said.

The study on public and private schools came out the week before Secretary of Education Margaret Spellings and Republican lawmakers unveiled a $100 million proposal to make vouchers for private school tuition available under the No Child Left Behind Act. In questioning the wisdom of that proposal, critics cited the NCES report showing public school students outperforming their private school peers.

Despite that debate, Mr. Schneider said that his concerns were his own, and that Bush administration officials did not try to influence his views on the subject. “The NCES,” he said, “is quite insulated from that kind of political pressure.”

A version of this article appeared in the August 30, 2006 edition of Education Week as NCES Calls For Sticking To the Stats

Events

Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Data Webinar
Using Integrated Analytics To Uncover Student Needs
Overwhelmed by data? Learn how an integrated approach to data analytics can help.

Content provided by Instructure
Classroom Technology Webinar How Pandemic Tech Is (and Is Not) Transforming K-12 Schools
The COVID-19 pandemic—and the resulting rise in virtual learning and big investments in digital learning tools— helped educators propel their technology skills to the next level. Teachers have become more adept at using learning management

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School Choice & Charters Opinion The Kind of School Reform That Parents Actually Want
Parents' inclination to focus on solving specific problems rather than system change helps explain the appetite for novel school options.
3 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School Choice & Charters Opinion What Do Parents Look for When Choosing a School?
New polling sheds light on what a nationally representative sample of parents had to say on this question this summer.
2 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School Choice & Charters Virtual Charters in Hot Water Again. Accusations of Fraud Prompt $150M Lawsuit
Indiana officials seek to recoup more than $150 million they say was either wrongly obtained or misspent by a consortium of virtual schools.
Arika Herron, The Indianapolis Star
2 min read
Indiana's attorney general Todd Rokita speaks at a news conference on Sept. 16, 2020, in Indianapolis. Rokita filed a lawsuit against a group of online charter schools accused of defrauding the state out of millions of dollars Thursday, July 8, 2021.
Indiana's attorney general Todd Rokita speaks at a news conference on Sept. 16, 2020, in Indianapolis.
Darron Cummings/AP
School Choice & Charters How the Pandemic Helped Fuel the Private School Choice Movement
State lawmakers got a new talking point as they pushed to create and expand programs to send students to private schools.
8 min read
Collage showing two boys in classroom during pandemic wearing masks with cropped photo of feet and arrows going in different directions.
Collage by Gina Tomko/EducationWeek (Images: Getty)