WASHINGTON--In the most comprehensive study of its kind yet conducted, researchers from Boston College have found evidence to confirm the widespread view that standardized and textbook tests emphasize low-level thinking and knowledge and that they exert a profound, mostly negative, effect on classroom instruction.
The three-year, $1 million study, sponsored by the National Science Foundation and released here last week, found that only a handful of the questions on major commercially available tests in mathematics and science, as well as on tests included in textbooks, measured the kinds of conceptual knowledge and problem-solving abilities reformers say should be integral to instruction in those fields.
And a survey of more than 2,200 teachers, along with site visits in six cities, confirmed that the tests influenced instruction and that most of the effects were negative, the study found. This was particularly true in classrooms with large numbers of minority students.
“The results of this study confirm our worst fears,’' Luther S. Williams, the head of the N.S.F.'s education and human-resources directorate, said in a statement.
George F. Madaus, the director of the center for the study of testing, evaluation, and educational policy at Boston College and the study’s director, said the findings present a “depressing picture.’'
“If this doesn’t change,’' he said at a press conference here, “an inordinate amount of time, attention, and preparation will be given to the wrong domains in math and science, domains that are not reflecting the outcomes we want.’'
But the study drew an angry response from commercial publishers, who said that they have been revamping their products to match the proposed reforms.
“This study is worthless if it ignores an entire generation of books and tests that came out after 1991,’' Peter Jovanovich, the president and chief executive officer of the Macmillan/McGraw-Hill School Publishing Company, said.
“Either the report is sloppy,’' Mr. Jovanovich said, “or it is willfully wrong.’'
Three-Part Study
The report notes that, while policymakers have in recent years turned to tests to provide accountability and to drive instruction, there has been no comprehensive analysis of the effects of such policies on classrooms.
To provide such an examination, the N.S.F. in 1989 launched the new study, which was composed of three parts:
- An item-by-item analysis of the six most widely used standardized achievement tests, as well as a sample of textbook tests, in math and science in grades 4 and 8 and in high school.
The tests, with copyright dates from 1985 to 1990, included the California Achievement Test, the Comprehensive Test of Basic Skills, the Iowa Test of Basic Skills, the Survey of Basic Skills of Science Research Associates, the Stanford Achievement Test, and the Metropolitan Achievement Test.
The textbooks, which varied according to subject matter, were those published by Addison-Wesley Publishing Company; D.C. Heath Company; Holt, Rinehart & Winston Inc.; Houghton-Mifflin Company; Charles E. Merrill Publishing Company; Prentice-Hall Inc.; Scott, Foresman & Company; and Silver Burdett Company.
- A nationwide questionnaire administered to 2,229 math and science teachers in grades 4-12.
- Interviews with 199 math and science teachers and 90 building-level administrators in six urban districts around the country.
Content Is Skewed
The study found that the content of the tests, particularly in math, did not reflect the recommendations of reformers in the field.
Compared with the standards of the National Council of Teachers of Mathematics, it notes, the tests overemphasized number theory and number systems, and underemphasized probability, measurement, algebraic thinking, and geometry.
The science tests, meanwhile, were skewed toward the life sciences, which made up 44 percent of the items, and away from the physical sciences, which accounted for 29 percent of the items.
The tests also found that the overwhelming majority of test items, particularly in math, measured low-level skills and knowledge.
In math, it notes, 95 percent of the items on both types of tests sampled recall of information, computation, and the use of formulas and algorithms in routine problems, while only 5 percent measured higher-level skills.
In science, it found, three-fourths of the items on the standardized tests, and 90 percent of those in the textbook tests, tested students’ recall of facts and routine applications.
Moreover, it found, 92 percent of the standardized-test items and 95 percent of those in textbook final tests did not demand knowledge of scientific procedures.
“This lack of emphasis on procedural knowledge,’' the report states, “conflicts with the current emphasis on hands-on, real-world science.’'
Test Preparation Widespread
The survey portion of the study also confirmed that testing is pervasive in schools and has a strong influence on teacher practices.
About 85 percent of the elementary and middle school math teachers, and 48 percent of the high school math teachers, who responded to the survey reported that one or more standardized tests were required of their students; 28 percent to 60 percent of the science teachers said such tests were mandated.
The study also found that more than half of all teachers used textbook tests at least once a month, and that three-fifths to 90 percent of teachers used teacher-made tests that often. However, the report notes, the site interviews show that many teachers create their own tests by using textbook items or adapting them somewhat.
In examining test-preparation practices, the study found that half of the teachers teach test-taking skills and encourage students to work hard on the tests, and that half the math teachers and a third of the science teachers teach topics known to be on the test.
Half or more of the math teachers, moreover, said a mandated standardized test influenced them “some’’ or “a lot’’ to include topics they would not otherwise have taught, or to change their emphasis.
Such test-preparation practices were particularly pervasive in classrooms with large proportions of minority students, the study found.
Three-fourths of teachers with at least 60 percent minority students in their classes reported pressure from their districts to raise test scores, compared with 60 percent of those with 10 percent or fewer minority students.
“Given standardized tests’ emphasis on low-level knowledge and skills,’' the report states, “their influence on the instruction of teachers with high percentages of minority students appears to be educationally and socially significant.’'
Questionable Practices
The interviews at the six field sites offered additional evidence of the influence of tests on instruction.
Three-fifths of the teachers interviewed, for example, said mandated testing narrowed or fragmented the curriculum, limited the nature of thinking, or forced them to rush their teaching too much, among other negative effects.
Of the 16 percent to 17 percent who cited positive effects, the study notes, most said the tests improved “coverage’’ of the curriculum.
The results also suggest that teachers and administrators felt that the tests forced them to compromise their ideals about good teaching. In some cases, the study found, educators were guilty of questionable practices--such as assigning the “best’’ teachers to tested grades and suspending low-performing students during test days--that cast doubt on the validity of the test results.
Copies of the summary documents and technical reports related to the study, “The Influence of Testing and Teaching Math and Science in Grades 4-12,’' are available from the Center for the Study of Testing, Evaluation, and Educational Policy, 323 Campion Hall, Boston College, Chestnut Hill, Mass. 02167.
