John Jacob Cannell, the West Virginia physician whose controversial study found that most students score above the average on normed achievement tests, has raised new questions about two widely used assessments.
In a letter sent this month to U.S. Secretary of Education William J. Bennett, Dr. Cannell charges that the California Achievement Test and the Stanford Achievement Test may have been “dumbed down’’ over the past decade.
Like his earlier findings, contained in a report last December, Dr. Cannell’s new conclusions are based on his own research, and have drawn cautious responses from a number of academic experts.
The degree of difficulty in the reading-comprehension section of the 1985 CATfor 2nd and 3rd graders was a full grade level below that of the 1977 test, Dr. Cannell contends.
In addition, he states, an analysis of the Stanford test administered to 4th and 5th graders showed a “profound drop in expository-reading difficulty’’ between 1972 and 1982.
“If this is true,’' he writes, “it will have profound effects on the reading proficiency of American schoolchildren for years to come.’'
“Teachers use these tests to evaluate grade-level reading ability,’' he added in an interview last week. “These tests present a false reassurance of reading abilities.’'
Dr. Cannell suggested that the findings may help explain the seemingly “anomalous’’ results of the 1986 reading test of the National Assessment of Educational Progress. Those results, which showed inexplicably large drops in performance between 1984 and 1986, may in fact reflect actual declines in student abilities, he said.
Kathleen A. Bursley, vice president and counsel for Harcourt Brace Jovanovich Inc., the firm that publishes the Stanford test, last week declined to comment on Dr. Cannell’s allegations because the firm had not received a copy of his letter to Secretary Bennett.
But a spokesman for the McGraw-Hill Book Company, which publishes the CAT, denied his charges, saying that the two versions of its test had been judged to be equally difficult on two separate measures of readability.
“There was no difference in the difficulty levels between the 1977 and 1985 tests,’' said Elizabeth Russo, McGraw-Hill’s director of public affairs.
‘He Could Be Right’
Researchers familiar with Dr. Cannell’s analysis agreed that he lacked sufficient evidence to support his charge. They added, however, that he had identified a potential problem with such tests that warrants further study.
“At some level, he could be right,’' said Jane C. Conoley, associate professor of educational psychology at the University of Nebraska at Lincoln. “But it’s hard to tell whether he’s found the smoking gun.’'
Dr. Cannell acknowledged that he was not qualified to conduct a detailed analysis of the tests, and he asked Mr. Bennett to oversee such a study. He added that “such a sequential analysis has never been done, except by the test publishers.’'
“This should be done by the education community, not by a general practitioner in Beaver, W.Va,’' he added.
Chester E. Finn Jr., the Education Department’s assistant secretary for educational research and improvement, said last week that the department’s forthcoming “consumer guide’’ to standardized tests might include information on whether such tests have become easier or harder over time.
“It’s a reasonable thing to look into,’' he said.
Seeking an Explanation
Dr. Cannell’s previous study, published by Friends for Education Inc., the advocacy group he established in West Virginia, found that most elementary-school pupils nationwide score above the average on nationally normed standardized tests offered by commercial publishers.
This phenomenon occurs, the report noted, because students are measured not against other pupils taking the test at the same time, but against norm groups tested in earlier years. (See Education Week, Dec. 9, 1987.)
The study--which looked at the CAT, the Stanford test, and four other major tests--gained considerable media attention and prompted a meeting at the Education Department in February. During that meeting, test publishers and researchers responded to the report by noting that student achievement had in fact improved since the time the norms were set.
Those higher achievement levels, they said, made it possible for most students to perform above national norms.
But Dr. Cannell said he was unconvinced by that explanation, and hypothesized instead that the tests may have become easier over time.
Readability Levels
Dr. Cannell said he undertook his new study in an effort to test this hypothesis.
His analysis found that the 1972 Stanford Form A test, administered to 4th and 5th graders, contained 140 syllables and 4.6 sentences for every 100 words. Using the Frye readability index, a commonly used measure of the difficulty of a reading passage, Dr. Cannell found that the passages on the test required an average 8th-grade reading level.
By contrast, he found, the 1982 Stanford Form E, intended for same grade levels, contained 132 syllables and 7.55 sentences per 100 words. Because shorter sentences with smaller words are easier to read, he said, the index indicated that the difficulty of the passages had declined to an average 6th-grade level.
In addition, Dr. Cannell said, the newer test appeared easier on casual examination. For example, he said, it included a reading passage from a television guide.
But, Dr. Cannell noted, the new test also required students to perform better than the earlier version. In the 1972 test, students needed 33 correct answers on 72 questions to be at the 50th-percentile rank; on the 1982 test, 39 correct answers on 60 questions were needed.
The newer edition of the CAT also required a lower reading ability, Dr. Cannell concluded. The 1977 CAT Form C, administered to 2nd and 3rd graders, contained 132.2 syllables and 10.35 sentences per 100 words, yielding an average readability at the 4th-grade level.
By contrast, he said, its 1985 version, CAT Form E, contained 125.8 syllables and 11.05 sentences per 100 words, and its average readability was at the 3rd-grade level.
But unlike the Stanford test, Dr. Cannell said, the new CAT did not require students to score substantially higher to remain at the 50th percentile. On the 1977 test, students needed 22 correct responses on 27 questions to attain that level; on the 1985 test, 29 correct responses on 35 questions were needed.
Researchers’ Response
But some scholars said that Dr. Cannell had failed to prove that the newer tests were easier.
“Just because the difficulty of a stimulus reading passage may have changed does not necessarily mean that the test was made easier,’' said Walter M. Haney, professor of education at Boston College. “It depends on the questions you ask.’'
In addition, noted Ms. Conoley of the University of Nebraska, even if the reading-comprehension section of a test is easier, the overall test may be equally difficult. Other sections, including mathematics, social science, and science, may have become more difficult over time, she said.
But both researchers called for further study of the issue.
Dr. Cannell’s conclusions “may in fact be the case,’' said Mr. Haney. “We don’t know. I hope somebody will pursue it.’'
