Two studies of the first program in the nation that spends public dollars on vouchers for poor children to attend private schools have reached dramatically different conclusions about whether it raises student achievement.
Both sides in the bitter and politically charged debate over school choice have seized on the conflicting analyses of the 6-year-old Milwaukee program. And the senior authors of the two studies have publicly criticized each other with unusual rancor, in part because of the high stakes.
The controversy pits a University of Wisconsin political scientist who has studied the Milwaukee program since its inception against researchers from Harvard University and the University of Houston who published their analysis last month.
The newer study concludes that students who were enrolled in the program for at least three years outperformed a comparable group of children who remained in the Milwaukee public schools.
The earlier research found “mixed results” and no significant gains in test scores. It has been widely cited by voucher opponents as evidence that the program doesn’t work.
Education experts said last week that neither study contains the conclusive, long-term evidence needed for making broad policy decisions.
“It’s going to take a lot more analysis than this to figure out what’s going on in these choice programs,” said Richard Elmore, a professor at the Harvard graduate school of education in Cambridge, Mass.
Charged Issue
The new research comes amid heightened interest in school vouchers nationwide. Republican presidential nominee Bob Dole has said a federal voucher program would be his top education priority if elected.
Governors and legislators in several states are considering voucher plans and have been closely watching the Milwaukee program and a similar effort that began last month in Cleveland.
Unlike in Milwaukee, however, Cleveland’s program includes the use of private religious schools, despite attempts by opponents to persuade the courts to stop it. (Please see “Court Clears Cleveland’s Voucher Pilot,” August 7, 1996.)
A 1995 Wisconsin state law that would open the Milwaukee program to religious schools is on hold while the courts decide whether it is constitutional.
“There are enormous amounts at stake,” said John F. Witte, the University of Wisconsin-Madison researcher who led the initial study of the Milwaukee program. “I certainly would appreciate the tone and the rhetoric to come down.”
Harsh Criticism
His analysis, conducted at the request of the Wisconsin education department, came under harsh criticism last month.
In an Aug. 14 article in The Wall Street Journal, Paul E. Peterson, the director of Harvard University’s program in education policy and governance, and Jay P. Greene, an assistant professor of political science at the University of Houston, wrote that Mr. Witte’s study was “so methodologically flawed as to be worthless.”
Mr. Witte shot back last week that the study by Mr. Peterson and his colleagues was a “confusing, tortured effort” to find evidence that choice works.
The new study found that students in their third and fourth years in the Milwaukee choice program scored 3 and 5 percentage points higher in reading, on average, than students who applied to the program but were not selected and remained in the public schools. The voucher students averaged 5 and 12 percentage points higher in math, Mr. Peterson and his colleagues report.
“If similar success could be achieved for all minority students nationwide, it could close the gap separating white and minority test scores by somewhere between one-third and one-half,” the researchers write in their study.
Mr. Peterson testified in court last month in support of expanding the voucher program to religious schools.
The five annual reports issued by Mr. Witte and his colleagues between 1991 and 1995 found no such improvement.
Although parents who sent their children to private schools under the plan were more satisfied and more involved than their public school counterparts, the participating students’ test scores neither improved nor differed significantly from those of other Milwaukee students, the reports concluded.
How could two studies of the same program reach such divergent conclusions?
“These are radically different studies, and I think they were done for different purposes,” Mr. Elmore said.
Control Groups Vary
The studies contain several key differences, notably in the selection of a control group against which the choice students were compared.
Mr. Witte and his colleagues compared data for students in all 12 of the private schools in the program with scores both from Milwaukee public school students as a whole and from the district’s low-income students.
They used statistical analysis to account for differences between the choice-program students and the control groups that might have influenced test scores, such as race, income, and sex.
But Mr. Peterson and his colleagues contend that some of the differences between the groups were so substantial that even with such controls, those studies “compared apples to oranges.”
Their research compared voucher students with children who applied for the program but were not selected. They believe this analysis produced more accurate results.
Students applied to the program each year for a seat in a particular grade in a particular school.
If there were more applicants than seats available, students were chosen at random. This, the researchers say, created two ideal experimental groups--the selected and nonselected students.
Theoretically, education researchers say, this is a much stronger design.
But the new study looked at a far smaller slice of students than the earlier research did. By the program’s fourth year, test data in math were available for only 110 selected and nonselected students, out of more than 1,000 children who have participated in the program.
The small numbers were caused by the program’s high attrition rate, which averaged 30 percent a year. In reading, scores were available for only 108 students.
The small samples, said Mr. Elmore of Harvard, make it hard to draw broad conclusions from the findings.
Measuring Gains
The studies also differed in the way they looked at test scores.
Mr. Witte’s analyses compared changes in test scores for groups of students from one year to the next. But Mr. Peterson asserts that such an analysis, which controls for the previous year’s test scores, masks the improvements in those scores that the program was designed to create.
His team, operating on the theory that students need time to adjust to a new learning environment, looked at the performance of individual students over time. They found that choice students did not outperform nonselected students in the program’s first two years, but that those who remained in the program at least three years pulled ahead of the control group.
Attrition Problem
The biggest problem for both teams of researchers was the high attrition rate.
“You just don’t know who those children are who don’t come back,” said Jane A. Stallings, the director of the Center for Collaborative Learning at Texas A&M University in College Station and a past president of the American Educational Research Association. “That really skews all of Peterson’s data and is very damning, I think.”
The two teams of researchers made different assumptions about the students who left the program.
Mr. Peterson and his colleagues cite data showing that the students who remained in the program and those who left after two years had similar scores at the end of the second year. They argue that there was little academic difference between those who stayed and those who left.
But Mr. Witte maintains that the students who left the program were, on average, those with lower prior test scores. Those who stayed in, he argues, were a select group.
In a similar way, the researchers differ over the comparison group of nonselected students. Mr. Witte asserts that nonselected students for whom test data were available came from poorer, less educated, less involved families than the nonselected students who left the school system.
‘Cautious, Encouraging Sign’
Experts said last week that the Peterson study was potentially important, but added that they lacked adequate information about its methodology to draw firm conclusions.
“It’s evocative. He may actually be correct,” said Peter Cookson, the director for educational outreach and innovation at Teachers College, Columbia University, in New York City. “But the study that he conducted is extremely limited, and it’s not a good idea to base large-scale policy decisions on it.”
Christopher Jencks, a professor of public policy at the John F. Kennedy School of Government at Harvard, agreed. “I think they’ve done it right, as far as I can tell from what they report,” he said.
But he noted that more than 80 percent of students in the choice program attended one of three private schools. So what the study essentially showed, he said, is that those three schools are more effective than the Milwaukee public schools.
“It’s kind of a cautious, encouraging sign,” Mr. Jencks said. “But it does not say we should do things totally different. We just don’t have that kind of evidence from this tiny number of schools and kids.”
