Chapter 1 Testing: It's No 'Field of Dreams'
In the RAND report "Federal Options for Improving the Education of Low-Income Students,'' released last week, we recommend reformulating Chapter 1 to play a far more significant role by increasing and concentrating funding for the nation's lowest-income schools. (See story on page 22.)
The program would then have the potential to go beyond remedial instruction for relatively few students and, instead, provide comprehensive improvements in the overall quality of education in our poorest communities. For reasons described below, we also conclude that current Chapter 1 testing requirements should be modified.
Conventional wisdom holds that we can test our way to school improvement--perhaps best described as a "Field of Dreams'' argument: Build a test and they will learn. Indeed, testing of students currently permeates virtually every aspect of the Chapter 1 program. Students are tested first to determine program eligibility and, at the end of the year, to see how much they have learned. Policymakers hope that the more they hold schools accountable for the test scores of Chapter 1 students, the more their educational programs will improve.
RAND invited comment from a diverse group of educators, policymakers, and researchers to assess the impact of Chapter 1 testing requirements. Few of the respondents had anything positive to say about current testing practices. The proliferation of testing has led to a complex set of problems and negative incentives:
- The testing encourages the teaching of a narrow set of measurable skills that often have little to do with what educators and parents value most. The mandated tests--and the rote learning associated with them--are particularly common in classrooms with high proportions of low-income and minority children.
- The use of test scores for funds allocation typically results in less funding for the schools that make achievement gains. The reliance on test scores, therefore, works against schools that have strong programs in the early years or promote successful students out of Chapter 1. If they succeed, as defined by the test scores, they lose money.
- Test-score differences from year to year, or from school to school, tell little about the quality of the educational program. The quality of an education system, of an individual school, or of a specific program--for example, Chapter 1--cannot be measured simply by comparing test-score fluctuations from one year to another, or by comparing schools or classrooms on test scores. The reason is that the results do not control for changes in student population, incentives for encouraging certain students to take (or not to take) the test, or the consistency (or lack of it) between the test and the instructional program.
The current Chapter 1 testing requirements do not lead to improvements in education. They tell us only what we already know--the effects of inadequate resources and poverty on the learning experience.
It is sometimes argued, however, that testing can be improved by developing innovative new tests, called "authentic tests,'' which would include performance assessments, essay exams, and portfolio assessments. Little attention is paid to how long such tests would take to develop, how much they would cost, and whether they could be administered on a large scale. "Authentic assessment'' for all Chapter 1 schools does not now exist. It would be expensive to develop and administer, although it might be useful for research or diagnostic purposes in individual schools.
Tests are also used to identify poorly performing schools so that school districts and states can intervene. The 1988 Hawkins-Stafford amendments to Chapter 1 added new provisions to encourage program improvement and greater accountability. Chapter 1 programs deemed to need improvement are those in which aggregate achievement scores of participating students show either no change or a decline over the course of a year. School districts are required to intervene to upgrade performance in such schools. Following district intervention, states are authorized to help design and implement joint state-district improvement plans for schools that continue to show no improvement.
Unfortunately, the tests that determine the need for program improvement are inherently unreliable and therefore not well suited for the intended purpose. In the nationally representative Chapter 1 Implementation Study, about one-half of the schools identified as needing help "tested out'' of program improvement in the second year without making any changes in their Chapter 1 programs. The scores simply improved depending on a whole variety of circumstances that could not be identified. Test scores tend to fluctuate so much from year to year--apart from changes in the quality of education--that many schools identified as requiring program improvement apparently did nothing but wait until the next testing period, successfully counting on testing out of the requirements.
These findings do not mitigate the importance of district or state assistance to "failing'' schools. They do, however, point out the impracticality of federally mandating this intervention nationwide based on scores on standardized tests. Educators and parents know well that the measure of the success or failure of a school goes well beyond test-score fluctuations from year to year. It includes such factors as (1) the overall school environment and the clarity of its mission; (2) the school's capacity for problem identification and resolution, as shown by the responsiveness of its educational program to the identified needs and problems; and (3) broad indicators of student performance and progress, for example, grades, attendance, promotions, high school graduation, and college attendance.
In short, the evidence from both research and practical experience suggests that federal testing requirements contribute more to bureaucracy, paperwork, and costs than they do to the quality of education. This conclusion also applies to recent proposals to increase Chapter 1 accountability requirements as a trade-off for reducing other regulations. The fact is that these proposals cannot be implemented without continuing to incur the negative consequences of current testing practices.
We recommend, therefore, that federal requirements for Chapter 1 testing--either for purposes of accountability or for determining school or student eligibility for program participation--be eliminated. Chapter 1 students should take the same tests routinely given to other children in their school district.
In reality, no testing program can separate the effects of Chapter 1--currently 3 percent of total funding for elementary and secondary education--from the effects of either the overall education experience or the broader environment. School districts and states have many pressures for educational accountability; the choice of specific measures can best be left to local discretion. Federal testing requirements, if eliminated for program accountability and student eligibility, would cease to drive the educational program in low-income schools, encourage the teaching of a narrow set of skills, or create perverse incentives that punish schools for raising achievement.
We should also stop pretending that testing requirements produce information that is useful to federal policymakers. Anyone who has tried to collate and interpret the test-score submissions from school districts throughout the country knows that these tests have merely created administrative burdens and tell us little about the quality of Chapter 1 programs.
Information about the effectiveness of federal education programs should continue to come from long-term, focused research that provides a more general sense of trends in the education of low-income students. The best of this research has served the education community well in the past and can be expected to continue to provide essential information about both the effectiveness of Chapter 1 and the broader education of low-income children.
Iris C. Rotberg is a senior social scientist with the RAND Institute on Education and Training. She directed the study of federal policy options for improving the education of low-income students.