When Sherrie L. Matula took all of her test-preparation materials for the Texas student assessment and stacked them atop one another, they stood about as tall as she is—close to 6 feet.
That pile of materials illustrates the high-stakes testing fire licking at teachers’ feet at Maesmythe Elementary School in Pasadena, Texas, where Ms. Matula teaches 4th grade. Besides the state’s heavy emphasis on exam-driven accountability, her district bases teachers’ salaries in part on student test scores. Those policies have forced teachers to teach to a basic-skills test, Ms. Matula says, echoing an increasingly common complaint of educators in many states.
But a forthcoming study by researchers from the University of Wisconsin suggests there may be little overlap between what state assessments test and what teachers teach. In one of the 10 states studied, the overlap in one subject was found to be just 5 percent.
The findings raise questions about the implementation of a basic premise of standards-based accountability: States should delineate what students should know and be able to do, teachers should match instruction to those standards, and state tests should measure how well students meet those expectations.
“It’s a very sloppy system right now,” said John F. Jennings, the director of the Center on Education Policy, a Washington research group. “Teachers are teaching what they think is best, and that might not be aligned” with the state standards or assessments.
The researchers at the Wisconsin Center for Education Research, based at the University of Wisconsin-Madison, surveyed mathematics and science teachers and examined state-administered exams in those subjects for grades 4 and 8. The detailed survey instrument asked teachers about the content they teach and the classroom practices and modes of thinking that their instruction requires.
Because they were not chosen at random, the 600 teachers surveyed are not statistically representative of teachers in those states, but the data do offer some insight into the effects of standards-based accountability. State education departments helped the researchers line up volunteers for the survey.
Few Performance Items
Instruction across the 10 states studied was similar, according to the researchers. But the content that the state exams assessed varied greatly.
The tests, the researchers found, mainly asked students to recall facts and perform other simple procedures, while classroom instruction focused more on solving novel problems or applying skills.
The highest level of overlap between teaching and the test was 46 percent—in 4th grade science in a single state. The lowest level, 5 percent, between a state test and instruction was in 8th grade math.
States don’t test students on higher-end expectations because it is too expensive, said Andrew C. Porter, the principal researcher and the director of the center.
For instance, the study found that one-fourth of science-class time was spent on hands-on science or laboratory activities. Yet such performance items were rarely included on state tests.
The Council of Chief State School Officers helped the researchers gather their data, and the National Science Foundation subsidized the project, “Using Data on Enacted Curriculum in Mathematics and Science.”
The report, which was previewed at this year’s American Educational Research Association conference in April, will be released this summer.
The participating states were Kentucky, Louisiana, Massachusetts, Minnesota, Missouri, North Carolina, Ohio, Pennsylvania, South Carolina, and West Virginia. State-specific results will not be released.
‘Doing More’
Sharon M. Stenglein, a math specialist in the Minnesota education department, agreed with the study’s conclusion that the type of test makes a big difference in the quantity and quality of alignment.
“Our minimum-competency test had a narrow range of content and cognitive demand. ... Teachers [in our state] are doing more,” she said. “That would show a low alignment, and that’s fine.”
Mr. Porter said that not all states considered a low level of alignment acceptable. When he presented his findings to them, many said they would revise their exams to better reflect teacher practice, he said. Each state, he noted, has to find the level of alignment that best suits its needs.
The lack of cohesion between instruction and tests was not a surprise to some observers.
“This is a snapshot of one stage [of standards-based changes], and it’s a pretty early stage,” Mr. Jennings said. “I’m not surprised that teachers are not teaching to the test.”
Other experts raised methodological concerns, in addition to the nonrandom selection of the teachers surveyed.
“The complexity of the teaching process is hard to summarize using something as simplistic as a survey,” said Daniel P. Mayer, a researcher at Mathematica Policy Research Inc., based in Washington. Mr. Mayer, who had not read the study, has investigated the differences between what teachers say in surveys and what they do in the classroom. He has found that teachers often think of themselves as practicing a certain innovative method, when in fact their instruction is not much different from traditional approaches.
Regardless of the reasons for the high degree of misalignment between teaching and testing the Wisconsin researchers found, some critics have attacked states for not doing enough to explain to teachers what is on state tests—an omission seen as especially troubling given the penalties and rewards states increasingly have attached to their exams.
“States are afraid to dictate [specific standards], but then issue the darn tests,” Mr. Jennings said. “Then, they go to the press saying schools are failing and teachers are not doing their job. ... That’s not fair.
“Teachers should be told [by the state] that this is the instruction that we expect if you want your students to do well on these tests.”
|
John L. Smithson, a project director at the Wisconsin center, said that for the most part, states have used high-stakes tests rather than state academic standards to guide instruction.
“The tests are attractive [to states] because they get people’s attention, and they provide a tool to measure outcomes,” he said. “It’s an attractive policy but limited.”
The research group has not yet examined state standards using the same methods it employed to look at instruction and assessments. That is the next step in the project.
Teaching to the Content
Some educators say they do not feel much state pressure to teach to the test.
Ethel C. Williams, an 8th grade language arts teacher at the 71st Classical Middle School in Fayetteville, N.C., said that her instruction is “student-focused and [state standards] focused.”
“I would not like to know what is on the test. Then I would teach to the test and not to the students,” she added.
Ms. Williams acknowledged that she does teach students some test-taking strategies, including such practices as perusing a question before reading the accompanying reading passage. That helps students focus on what is being asked in a particular test question, she said.
W. James Popham, an assessment expert and a professor emeritus at the University of California, Los Angeles noted that such test-preparation practices can enter into a gray area between teaching to actual test items and teaching to the content represented by the test.
“Familiarity with the form of the test, that’s good,” he said, “but we find, in some schools, that they are shutting down everything for a month or more” to prepare for the test.