Using off-the-shelf tests for standards-based assessment is a “risky business,” according to a forthcoming study conducted for the California education department.
|
“The vast majority of items on these tests have very little to do with the California 8th grade math standards,” said the study’s lead author, William H. Schmidt, a professor of educational statistics at Michigan State University. “It’s just basic, pedantic arithmetic, rehashed again and again,” asserted Mr. Schmidt, who presented his preliminary findings at the National Conference of Large-Scale Assessment here last week.
The California department commissioned the Third International Mathematics and Science Study Center, where Mr. Schmidt is the national research coordinator, to examine the content of off-the-shelf math tests. The methodology for the project was first used in the TIMSS assessment and codes the academic content of test items as well their degree of rigor.
Researchers examined about 75 norm-referenced, off-the-shelf tests for grades 2-8, including series from the Stanford Achievement Test, the Comprehensive Test of Basic Skills, and the Iowa Tests of Basic Skills. The report is scheduled to be published in the fall.
While Mr. Schmidt looked only at California standards for this latest study, he has examined many other states’ standards and how they line up against off-the-shelf tests.
The California study found that elementary and middle school off-the-shelf mathematics tests do not adequately measure whether students have mastered the state’s math standards, or most other states’ standards, because they rely heavily on basic arithmetic.
For example, the 8th grade math standards for California make no mention of whole-number operations. Yet, about 25 percent to 80 percent of the off-the-shelf test items for that grade focused on simple whole-number computations such as addition and multiplication.
In addition, the California 8th grade standards emphasize higher-level concepts such as advanced-number theory and congruence and similarity. But such concepts were rarely tested. Even in the 8th grade exam with the most coverage of those more complex topics, only about 5 percent of the test dealt with any one of the topics, the study found.
Test Misuse?
Gerry Shelton, who oversaw the project for the state education department, called the use of off-the-shelf tests for measuring student understanding of content standards or for high-stakes accountability purposes “post-hoc planning.” Too often, he said, states put tests in place and then policymakers endow them with different uses. Norm-referenced tests should be used only to compare students against national averages, said Mr. Shelton.
Joanne M. Lenke, the president of Harcourt Educational Measurement, which publishes the Stanford Achievement Test, agreed that such ready-made tests could be misused.
Off-the-shelf tests are “not developed to reflect any particular state standards,” she said, but rather apply “a national-consensus curriculum.” Specifically, she said, her company uses a variety of textbooks, state standards, and national standards to construct tests.
Norm-referenced tests “reflect what is being taught in 8th grade mathematics classrooms across the country,” Ms. Lenke said.
Breaking the Trend
The study found closer alignment with off-the-shelf math exams and randomly selected California district standards than with the state’s standards. Mr. Schmidt theorized that the alignment could be due to such factors as test publishers heavier marketing of districts than states.
Critics have historically contended that standardized tests measure a low level of knowledge.
“Standardized tests are often quite limited in depth,” said Christopher T. Cross, the president of the Council for Basic Education, a Washington-based organization that advocates high academic standards. “Standards-based assessments are trying to break this trend of just testing basic math, because when you have clear standards that go beyond the basics, this will drive new assessments and raise the level of instruction and testing.”
Viewing the assessments in an international context, Mr. Schmidt said, “the top-achieving countries on TIMSS all end basic arithmetic instruction around the 4th grade.” Many of those higher achievers instruct schoolchildren on proportionality and percentages in the 5th and 6th grades as the transition into the fundamentals of algebra and geometry begins, he said. The inertia toward standards-based tests in the United States, he added, means that traditional psychometric methods are used to shape off-the-shelf tests into a lump of basic, arithmetic items.
In an effort to move away from off-the-shelf tests, California administered for the first time this year a more customized standards-based version of the Stanford Achievement Test-9th Edition in math and English language arts and an off-the-shelf exam. Mr. Schmidt’s analysis was not applied to the customized math exam.
The state next year will administer standards-based assessments in history and science.
