Although the Education Department’s research centers and regional laboratories have produced some high-quality research, a significant portion of their output “could be substantially improved,’' a study commissioned by the agency’s former research head has found.
A draft of a report by Maris A. Vinovskis, a professor of history at the University of Michigan, cites several exemplary studies by the institutions, including an analysis of youth-training programs, the development of the “Success for All’’ program for disadvantaged students, and “pioneering’’ work in understanding how children learn mathematics.
But the unreleased analysis, a copy of which was obtained by Education Week, argues that other studies are marred by design and methodological problems.
And over all, the report contends, the research work at the labs is “disappointing.’'
Moreover, the department’s office of educational research and improvement, which oversees the centers and labs, is ill equipped to monitor the quality of their work, the report suggests.
“The department is spending too little on research, and what little it spends is not well-spent,’' said Diane S. Ravitch, the former O.E.R.I. head who commissioned the report.
Emerson J. Elliott, the acting assistant secretary for educational research and improvement, responded in an interview that the report, which is expected to be completed this summer, has already spurred improvements in the agency. The O.E.R.I. also will consider other changes, such as developing standards for the research work it funds, he added.
“Maris has affected the way people talk around here,’' Mr. Elliott said.
But Mary Kennedy, the chairman of a group of directors of the research centers, said Mr. Vinovskis’ study was flawed by a narrow view of what constitutes high-quality research.
“He raised a lot of important questions, such as how O.E.R.I. can make sure the quality is good,’' Ms. Kennedy said. “I think it’s important that O.E.R.I. do that.’'
“It is unfortunate that they decided to do it with one person making his own judgments, one not even knowledgeable in all areas’’ the centers and labs study, said Ms. Kennedy, the director of the National Research Center on Learning To Teach at Michigan State University.
‘Works In Progress’
The study by Mr. Vinovskis is the latest to examine an agency that has come under increased scrutiny from inside and outside the government.
Last year, the National Research Council recommended a major overhaul of the O.E.R.I. (See Education Week, April 8, 1992.)
In addition, the House passed legislation that would have expanded and restructured the agency. Although that measure died at the end of last year’s session, it is expected to be reconsidered this year.
Gerald E. Sroufe, the director of governmental and professional liaison for the American Educational Research Association, said he hopes Mr. Vinovskis’ report does not divert attention from the N.R.C.'s report, which was much broader in scope and represented the judgment of a panel of experts.
But Mr. Vinovskis said his aim was more narrow. Ms. Ravitch, the former assistant secretary for educational research and improvement, originally asked him to conduct a study that would look only at the quality of research conducted by the agency. Because of time limitations, moreover, he was able to examine just the centers and labs.
“These are works in progress in an overall view of the agency,’' said Mr. Vinovskis, who is scheduled to return to the University of Michigan in July.
History, Research Examined
To conduct his study, Mr. Vinovskis examined the history of the centers and labs, which were created during the Johnson Administration to provide systematic study of educational issues and to disseminate research-based innovations to schools.
In addition, Mr. Vinovskis also conducted an in-depth analysis of the work of five of the 23 research centers: the Center for Research on Effective Schooling for Disadvantaged Students, based at Johns Hopkins University; the Center for Research on Evaluation, Standards, and Student Testing, based at the University of California at Los Angeles; the National Research Center on Student Learning, based at the University of Pittsburgh; the National Center for the Study of Writing and Literacy, based at the University of California at Berkeley; and the Policy Center of the Consortium for Policy Research in Education, based at Rutgers University.
Mr. Vinovskis also studied the work of two former centers, the National Center for Improving Science Education and the National Center on Education and Employment.
In addition, he examined five of the 10 labs: the Regional Laboratory for Educational Improvement of the Northeast and Islands; the Southwest Educational Development Laboratory; the Mid-Continent Regional Educational Laboratory; the Northwest Regional Educational Laboratory; and the Far West Laboratory for Educational Research and Development.
Mr. Vinovskis acknowledged that his sample was not representative, but said a review of materials from institutions he did not closely study convinced him that the others did not differ substantially.
“Nothing I saw in that made me nervous about my samples,’' he said.
‘Rather Modest’
As previous studies have shown, Mr. Vinovskis notes that funding for the research centers and labs has dropped sharply in the past three decades.
In constant dollars, the funding for the average center--about $1.2 million in fiscal 1992--is about a third of what it was 25 years ago, the study indicates. The average lab budget, $2.3 million, is half its inflation-adjusted peak level in 1970.
Only 52.9 percent of the already-reduced funds in the centers actually went toward research, Mr. Vinovskis found, and those funds were spread over a number of studies. The rest of the money went toward overhead, administrative expenses, and dissemination.
“As a result,’' the draft report states, “the total annual amount of money directly available for any given project was rather modest and in effect usually precluded any large-scale research undertakings.’'
The labs, whose mission is broader, spend a smaller proportion of their funds on research --about a third, the report says.
Examining the type of research the institutions conduct, Mr. Vinovskis found that the five centers studied placed relatively little emphasis on basic research, which got 18.7 percent of the funding, and on development, which got 13.5 percent. And he suggests that some of the work is limited to reviews of existing research, rather than new studies.
A Mixed Grade
In the most controversial part of the analysis, Mr. Vinovskis concludes that the quality of the work is mixed. This is particularly true of the labs, he said, citing only Far West as doing exemplary work.
But, he added, that finding may surprise some policymakers, who have contended that the institutions are uniformly poor.
“Too often this has become a political football--'Are you for or against centers and labs?’'' Mr. Vinovskis said. “Anybody who looks candidly will find evidence of first-rate work, and of problematic work.’'
Among the studies cited in the report as problematic are those that used small or unrepresentative samples, such as a Berkeley study that looked at “a few’’ students and a Rutgers study that examined 48 teachers in four cities.
But sample size is only one of many criteria for evaluating the quality of a research project, Ms. Kennedy of Michigan State contended. Others include the quality of the question asked, the opportunities to find disconfirming evidence, and the methods for documenting the validity of the study, she said.
“Ask a researcher to define for you what makes a study good, you’ll get a half-dozen criteria, none of them sample size,’' she observed.
Ms. Kennedy also said that literature reviews are an important type of research work, and that the proportion of basic research conducted at the centers is higher than Mr. Vinovskis indicated.
Dena G. Stoner, the executive director of the Council for Educational Development and Research, a Washington-based trade association of R&D institutions, including the labs, said that research represents a small fraction of the work of the labs. In fact, she said, the Education Department in the 1980’s prohibited the labs from conducting research, preferring that they spend their time and money on development, dissemination, and technical assistance to schools.
“The institutions are whipped around by continually varying sets of criteria for success,’' she said.
