Assessment and the 'Educational Impact Statement'
The testing explosion of the past two decades has not produced the school improvement its proponents promised. The only clear result is that U.S. students are now the most heavily tested in the world. More than 100 million standardized tests are now administered in our classrooms each year.
Despite this massive testing, there is less public oversight and regulation of the tests administered to our children than of the food we feed our pets. In no other public area of society in which life-determining decisions are made is regulation so completely absent.
Standardized exams are used for a bewildering array of important purposes, from screening and sorting to grade promotion and high school graduation. Their fill-in-the-bubble limitations, however, have led to dumbed-down curricula, narrowed learning, and discriminatory tracking.
To be sure, this negative experience has largely been with low-level, multiple-choice tests. Indications do exist that careful use of high-quality performance-based assessments can induce positive changes in students, teachers, and classrooms. But there is still no hard data demonstrating that these improved assessments, when expanded to large-scale, high-stakes use, will have a similarly positive impact, especially if this expansion is done hastily and with high stakes for students and teachers.
In spite of this weak track record, testing advocates are calling for a national assessment system as a crucial component of what they call education reform. Whether or not a national assessment system is instituted, it is likely that large amounts of testing will continue for at least the near future. For example, some states are developing several different performance assessments. Even if these partially replace multiple-choice tests, rather than just add to them, the amount of testing, or assessing, may end up increasing.
The United States needs a more cautious and socially responsible approach to assessment. It is also imperative to do whatever possible to transform assessment into a tool for helping individuals and education systems improve. High-quality assessment, evaluation, and feedback are necessary components of high-quality education.
Based on experience, the United States cannot afford to remain dependent on corporate or governmental test makers' promotional claims about the quality of their products, no matter what type of assessments they propose. What is needed is a new process which provides a democratic check against the natural boosterism of testing proponents to protect against the possible damaging consequences of assessments.
One mechanism that is rapidly gaining support is development of an Educational Impact Statement process similar to the Environmental Impact Statement now required before any major federally-funded construction project. The impact statement would be designed to help public officials make decisions about assessment programs that are based on an understanding of potential consequences for students, families, schools, communities, and society. Its safeguards would help protect students and education systems from damage, and assure development of fair, high-quality assessments that enhance education reform.
The educational impact statement would be an integral part of the development of an assessment system. It would require education systems to compile a dossier of hard evidence, not just public-relations rhetoric, demonstrating how implementing their proposed plan would actually improve education.
Assessment users would have to show how their proposal meets specific standards, including:
- Data demonstrating that it is bias-free and flexible enough to enable a diverse population to demonstrate competence in a variety of ways.
- Validity studies proving that the assessment is appropriate for its specific intended purposes and congruent with the educational goals of the system.
- Assurances that the assessment will not be used as the sole factor in making high-stakes decisions about students, teachers, or schools, and that multiple forms of evidence will be used to assess all students.
- Justification for the amounts of on-demand assessment or testing to be conducted.
- School delivery and system-performance standards to hold public officials and institutions accountable for ensuring that all students are afforded an equitable opportunity to perform well.
- Provisions for staff and curriculum development to implement the new assessment, as well as public education plans to explain the plan to the community.
- Due-process protections for test takers and their parents, including disclosure requirements and the right to challenge questionable assessment results.
- Compliance with the National Forum on Assessment's "Criteria for Evaluating Student Assessment Systems,'' which has been endorsed by over 70 national and regional education and civil-rights organizations and which provides an outline of what an assessment system should and should not do.
An educational impact statement, or E.I.S., should be done at the appropriate level. For state assessment systems, there would be a state E.I.S. conducted by the state education department. For a district assessment plan, there would be a district-level E.I.S. conducted by the local education authority. If ever there is a national system of assessments, then the U.S. Education Department would conduct an E.I.S.
To prepare an impact statement, the education system would notify the public of its draft assessment proposal through aggressive outreach, with an emphasis on affected constituencies. It would then hold public hearings to solicit broad input from experts, including educators and psychometricians, as well as lay people such as parents, community activists, and business leaders. Written testimony on the proposed assessment would also be accepted.
Next, an assessment plan responding to the concerns raised during the hearings and comparing proposed alternatives would be compiled. Because information about the system will accumulate over time, the plan would also have to include benchmarks for evaluating its implementation and methods for its further refinement, using the standards listed above.
In the coming year, Congress will reauthorize the Elementary and Secondary Education Act, which includes Chapter 1. This program has been a primary cause of the testing explosion and has been severely damaged by its heavy testing requirements. As a result, many groups and individuals involved with Chapter 1 have called for completely overhauling its assessment procedures. Requirements for at least states, if not districts, to develop and implement impact-statement procedures should be included in the act's reauthorization as part of restructuring Chapter 1.
Ultimately, all assessment systems receiving federal funds should go through this process, thus enabling a national review and reform of assessment from the bottom up. Research on the development, implementation, and effects of the educational impact statement at the state level should be conducted by the Office of Educational Research and Improvement.
Just like tests themselves, new procedural requirements run the risk of causing more problems than they solve. Two possibilities for the E.I.S. deserve immediate consideration. One danger is that the process will become little more than a bureaucratic impediment to the development of good assessments. The goal is not to produce a paperwork obstacle course. Rather, it is to design an effective quality-control mechanism that relies on democratic participation to block poor proposals, improve weak ones, and build community support for solid plans.
A second potential problem lies with the difficulty of reaching consensus about what good assessments actually are. Indeed, our communities are diverse and often politically complicated. Under the current system, this diversity is hidden beneath the mask of test maker claims of their products' "objectivity.'' The Educational Impact Statement process will bring these crucial issues into the public arena. Democracy may not always be simple, but it is preferable to behind-the-scenes rule by unaccountable entities like test manufacturers.
In addition to these pitfalls, the impact-statement process could still face the problem of a lack of adequate independent technical analysis. There is currently no governmental or private agency that could produce reliable, independent technical analyses of assessment proposals. Without this information, the public could remain at the mercy of the test makers for important technical information.
This problem could be solved in several ways. For example, some federal agencies provide technical-assistance grants to give grassroots organizations access to scientific expertise in the Environment Impact Statement process. Or an independent technical group could be established to analyze tests and assessments.
No process can absolutely guarantee that educational improvement will result from a new assessment or that all test use will be fair and valid. But the educational impact statement will minimize potential dangers and increase the likelihood that fair, high-quality, socially responsible assessment systems are actually implemented. That is why the Educational Impact Statement concept has already been endorsed by more than two dozen national school-reform and civil-rights groups working together in the FairTest-initiated Campaign for Genuine Accountability in Education.
The Environmental Impact Statement has already played an important role in protecting our nation's air, land, and water. Extending the concept to the educational arena offers the best chance we have to insure that assessment reform actually leads to excellence and equity in U.S. schools.
Monty Neill is the associate director of FairTest, the National
Center for Fair & Open Testing, in Cambridge, Mass.