The National Research Council has questioned the validity of the two largest studies of bilingual education issued by the Education Department in recent years.
A panel convened by the N.R.C., an arm of the National Academy of Sciences, said last month that the studies were designed too poorly to be conclusive.
Alan L. Ginsburg, who supervised the studies and requested the N.R.C. review as director of the Education Department’s planning and evaluation service, defended the studies, saying the N.R.C. panel judged them by unreasonably high methodological standards.
The first study--which department officials had said was likely to quell much of the debate over bilingual education--found that the three most common bilingual-education methods were equally effective. (See Education Week, Feb. 20, 1991.)
The other tracked the progress of limited-English-proficient elementary-school students over five years and concluded that the type of instruction provided to them depended more on local conditions and resources than the pupils’ academic needs. (See Education Week, May 2, 1990.)
The N.R.C. panel said the studies lacked a firm conceptual grounding, did not have specific objectives, and were not designed to answer the basic policy questions that motivated them.
“The real problem with the studies we reviewed was that the questions were not asked in a way that would lead to the proper assessments,’' said Stephen E. Fienberg, who chaired the review panel.
No Clear Agreement
The Education Department asked the N.R.C. to review the two studies in October 1990.
The review panel concluded that the studies suffered from the absence of a clear agreement on the objective of bilingual education.
The lack of agreement, the report said, complicates the tasks of designing valid studies and of deciding which programs are the best.
“The studies do not license the conclusion that any one type of program is superior to any other nor that the programs are equally effective,’' the N.R.C. panel said.
“Even if one of the programs was definitely superior,’' the reviewers wrote, “the studies as planned and executed could well have failed to find the effect.’'
In spite of their limitations, some of the studies’ findings appeared to be consistent with the results of other studies, and they both supported the theoretical foundations of native-language instruction, the panel found.
Both studies, the report noted, suggested that, under certain conditions, primary-language instruction is important for second-language development in language arts and mathematics.
The panel advised the Education Department not to conduct other analyses of data from the studies because they would not lead to solid conclusions.
It recommended, instead, that the department conduct “more focused and theoretically driven studies’’ that clearly define different instructional approaches and examine them in specific contexts.
The N.R.C. panel faulted the studies for relying too heavily on elaborate statistical methods to overcome shortcomings in research design. It suggested that the department may have been better off if the money spent identifying programs that exemplify certain educational approaches had instead been spent to establish programs that would have allowed for valid experiments.
Split in Research Community
Mr. Ginsburg said the N.R.C. report illustrated a division in the educational-research community.
The department’s studies, he said, were conducted by researchers who believe it is often impossible and unethical for scientists to create the precise conditions necessary for scientifically valid experiments.
The N.R.C. panel members, he continued, appear to belong to a second school of thought that backs experimental research only.
The N.R.C. report, “Assessing Evaluation Studies: The Case of Bilingual Education Strategies,’' is available for $19 prepaid, plus $3 shipping, from the National Academy Press, 2101 Constitution Ave., N.W., Washington, D.C. 20418; telephone (202) 334-3313 or (800) 624-6242.
