Study Cites Need To Improve E.D. Research Efforts

By Robert Rothman — May 05, 1993 7 min read
  • Save to favorites
  • Print

Although the Education Department’s research centers and regional laboratories have produced some high-quality research, a significant portion of their output “could be substantially improved,’' a study commissioned by the agency’s former research head has found.

A draft of a report by Maris A. Vinovskis, a professor of history at the University of Michigan, cites several exemplary studies by the institutions, including an analysis of youth-training programs, the development of the “Success for All’’ program for disadvantaged students, and “pioneering’’ work in understanding how children learn mathematics.

But the unreleased analysis, a copy of which was obtained by Education Week, argues that other studies are marred by design and methodological problems.

And over all, the report contends, the research work at the labs is “disappointing.’'

Moreover, the department’s office of educational research and improvement, which oversees the centers and labs, is ill equipped to monitor the quality of their work, the report suggests.

“The department is spending too little on research, and what little it spends is not well-spent,’' said Diane S. Ravitch, the former O.E.R.I. head who commissioned the report.

Emerson J. Elliott, the acting assistant secretary for educational research and improvement, responded in an interview that the report, which is expected to be completed this summer, has already spurred improvements in the agency. The O.E.R.I. also will consider other changes, such as developing standards for the research work it funds, he added.

“Maris has affected the way people talk around here,’' Mr. Elliott said.

But Mary Kennedy, the chairman of a group of directors of the research centers, said Mr. Vinovskis’ study was flawed by a narrow view of what constitutes high-quality research.

“He raised a lot of important questions, such as how O.E.R.I. can make sure the quality is good,’' Ms. Kennedy said. “I think it’s important that O.E.R.I. do that.’'

“It is unfortunate that they decided to do it with one person making his own judgments, one not even knowledgeable in all areas’’ the centers and labs study, said Ms. Kennedy, the director of the National Research Center on Learning To Teach at Michigan State University.

‘Works In Progress’

The study by Mr. Vinovskis is the latest to examine an agency that has come under increased scrutiny from inside and outside the government.

Last year, the National Research Council recommended a major overhaul of the O.E.R.I. (See Education Week, April 8, 1992.)

In addition, the House passed legislation that would have expanded and restructured the agency. Although that measure died at the end of last year’s session, it is expected to be reconsidered this year.

Gerald E. Sroufe, the director of governmental and professional liaison for the American Educational Research Association, said he hopes Mr. Vinovskis’ report does not divert attention from the N.R.C.'s report, which was much broader in scope and represented the judgment of a panel of experts.

But Mr. Vinovskis said his aim was more narrow. Ms. Ravitch, the former assistant secretary for educational research and improvement, originally asked him to conduct a study that would look only at the quality of research conducted by the agency. Because of time limitations, moreover, he was able to examine just the centers and labs.

“These are works in progress in an overall view of the agency,’' said Mr. Vinovskis, who is scheduled to return to the University of Michigan in July.

History, Research Examined

To conduct his study, Mr. Vinovskis examined the history of the centers and labs, which were created during the Johnson Administration to provide systematic study of educational issues and to disseminate research-based innovations to schools.

In addition, Mr. Vinovskis also conducted an in-depth analysis of the work of five of the 23 research centers: the Center for Research on Effective Schooling for Disadvantaged Students, based at Johns Hopkins University; the Center for Research on Evaluation, Standards, and Student Testing, based at the University of California at Los Angeles; the National Research Center on Student Learning, based at the University of Pittsburgh; the National Center for the Study of Writing and Literacy, based at the University of California at Berkeley; and the Policy Center of the Consortium for Policy Research in Education, based at Rutgers University.

Mr. Vinovskis also studied the work of two former centers, the National Center for Improving Science Education and the National Center on Education and Employment.

In addition, he examined five of the 10 labs: the Regional Laboratory for Educational Improvement of the Northeast and Islands; the Southwest Educational Development Laboratory; the Mid-Continent Regional Educational Laboratory; the Northwest Regional Educational Laboratory; and the Far West Laboratory for Educational Research and Development.

Mr. Vinovskis acknowledged that his sample was not representative, but said a review of materials from institutions he did not closely study convinced him that the others did not differ substantially.

“Nothing I saw in that made me nervous about my samples,’' he said.

‘Rather Modest’

As previous studies have shown, Mr. Vinovskis notes that funding for the research centers and labs has dropped sharply in the past three decades.

In constant dollars, the funding for the average center--about $1.2 million in fiscal 1992--is about a third of what it was 25 years ago, the study indicates. The average lab budget, $2.3 million, is half its inflation-adjusted peak level in 1970.

Only 52.9 percent of the already-reduced funds in the centers actually went toward research, Mr. Vinovskis found, and those funds were spread over a number of studies. The rest of the money went toward overhead, administrative expenses, and dissemination.

“As a result,’' the draft report states, “the total annual amount of money directly available for any given project was rather modest and in effect usually precluded any large-scale research undertakings.’'

The labs, whose mission is broader, spend a smaller proportion of their funds on research --about a third, the report says.

Examining the type of research the institutions conduct, Mr. Vinovskis found that the five centers studied placed relatively little emphasis on basic research, which got 18.7 percent of the funding, and on development, which got 13.5 percent. And he suggests that some of the work is limited to reviews of existing research, rather than new studies.

A Mixed Grade

In the most controversial part of the analysis, Mr. Vinovskis concludes that the quality of the work is mixed. This is particularly true of the labs, he said, citing only Far West as doing exemplary work.

But, he added, that finding may surprise some policymakers, who have contended that the institutions are uniformly poor.

“Too often this has become a political football--'Are you for or against centers and labs?’'' Mr. Vinovskis said. “Anybody who looks candidly will find evidence of first-rate work, and of problematic work.’'

Among the studies cited in the report as problematic are those that used small or unrepresentative samples, such as a Berkeley study that looked at “a few’’ students and a Rutgers study that examined 48 teachers in four cities.

But sample size is only one of many criteria for evaluating the quality of a research project, Ms. Kennedy of Michigan State contended. Others include the quality of the question asked, the opportunities to find disconfirming evidence, and the methods for documenting the validity of the study, she said.

“Ask a researcher to define for you what makes a study good, you’ll get a half-dozen criteria, none of them sample size,’' she observed.

Ms. Kennedy also said that literature reviews are an important type of research work, and that the proportion of basic research conducted at the centers is higher than Mr. Vinovskis indicated.

Dena G. Stoner, the executive director of the Council for Educational Development and Research, a Washington-based trade association of R&D institutions, including the labs, said that research represents a small fraction of the work of the labs. In fact, she said, the Education Department in the 1980’s prohibited the labs from conducting research, preferring that they spend their time and money on development, dissemination, and technical assistance to schools.

“The institutions are whipped around by continually varying sets of criteria for success,’' she said.

A version of this article appeared in the May 05, 1993 edition of Education Week as Study Cites Need To Improve E.D. Research Efforts


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Data Webinar
Education Insights with Actionable Data to Create More Personalized Engagement
The world has changed during this time of pandemic learning, and there is a new challenge faced in education regarding how we effectively utilize the data now available to educators and leaders. In this session
Content provided by Microsoft
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Accelerate Learning with Project-Based Learning
Earlier this year, the George Lucas Educational Foundation released four new studies highlighting how project-based learning (PBL) helps accelerate student learning—across age groups, multiple disciplines, and different socio-economic statuses. With this year’s emphasis on unfinished
Content provided by SmartLab Learning
School & District Management Live Online Discussion Principal Overload: How to Manage Anxiety, Stress, and Tough Decisions
According to recent surveys, more than 40 percent of principals are considering leaving their jobs. With the pandemic, running a school building has become even more complicated, and principals' workloads continue to grow. If we

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education California Makes Ethnic Studies a High School Requirement
California is among the first in the nation to require students to take a course in ethnic studies to get a diploma starting in 2029-30.
4 min read
FILE - In this Jan. 22, 2020, file photo, Democratic Assembly members, from left, James Ramos, Chris Holden Jose Medina, and Rudy Salas, Jr., right, huddle during an Assembly session in Sacramento, Calif. Medina's bill to make ethnic studies a high school requirement was signed into law by California Gov. Gavin Newsom on Friday, Oct. 8, 2021. (AP Photo/Rich Pedroncelli, File)
Education California Requires Free Menstrual Products in Public Schools
The move comes as women’s rights advocates push nationwide for affordable access to pads, tampons, and other items.
1 min read
Tammy Compton restocks tampons at Compton's Market, in Sacramento, Calif., on June 22, 2016. California public schools and colleges must stock their restrooms with free menstrual products under a new law signed by Gov. Gavin Newsom, Friday, Oct. 8, 2021.
Tammy Compton restocks tampons at Compton's Market, in Sacramento, Calif., on June 22, 2016. California public schools and colleges must stock their restrooms with free menstrual products under a new law signed by Gov. Gavin Newsom, Friday, Oct. 8, 2021.
Rich Pedroncelli/AP
Education Florida to Dock School District Salaries for Requiring Masks
Florida is set to dock salaries and withhold funding from local school districts that defied Gov. Ron DeSantis' ban on mask mandates.
2 min read
Florida Gov. Ron DeSantis speaks, Tuesday, Sept. 14, 2021, at the Doral Academy Preparatory School in Doral, Fla.
Florida Gov. Ron DeSantis speaks, Tuesday, Sept. 14, 2021, at the Doral Academy Preparatory School in Doral, Fla.
Wilfredo Lee/AP
Education More Than 120,000 U.S. Kids Had Caregivers Die During Pandemic
The toll has been far greater among Black and Hispanic Americans, a new study suggests.
3 min read
FILE - In this Thursday, Sept. 2, 2021 file photo, a funeral director arranges flowers on a casket before a service in Tampa, Fla. According to a study published Thursday, Oct. 7, 2021, by the medical journal Pediatrics, the number of U.S. children orphaned during the COVID-19 pandemic may be larger than previously estimated, and the toll has been far greater among Black and Hispanic Americans. (AP Photo/Chris O'Meara, File)