In 1984, the Council of Chief State School Officers voted to establish a center to coordinate state testing and evaluation programs.
The State Education Assessment Center, headed by Ramsay W. Selden, began operating last fall. It is charged with developing the state-by-state comparisons of education that the chiefs approved in November at their 1985 annual conference.
The current budget for the center is approximately $500,000, including a grant from the Charles Stewart Mott Foundation to support planning for some of the more difficult indicators, like teacher quality, and a three-year contract with the federal Center for Statistics (at $409,000 per year) to examine federal education data for accuracy, comparability, completeness, and timeliness.
Before joining the center, Mr. Selden was assistant director of the “excellence indicators” division at the National Institute of Education. He was also on the staff of the National Commission on Excellence in Education from 1983 to 1984. Mr. Selden holds a Ph.D. in educational-research methodology from the University of Virginia.
He recently spoke with Assistant Editor Lynn Olson about the goals and activities of the new center.
Q Federal officials say that the primary role of the federal government in education is data collection. Given that, why is the center necessary?
A We need to work with the federal government and the federal government needs to work with states. It has got to be a joint effort.
There are fairly specific and relatively mundane problems that we can help the federal government straighten out. One of them is comparability. Even what one would think are fairly straightforward statistics on education-like student attendance or completion of school-are complicated by the fact that states use different definitions and procedures to collect those data and report them to the government.
One of the services we can provide is to coordinate between states and the federal government in getting more comparable, more timely data.
There are other things the Center for Statistics [formerly, the National Center for Education Statistics] just does not do that the states really have to agree to do together, and which our center, as a collective effort by the states, gives them a vehicle for doing.
For example, the Center for Statistics does not have a program that would allow for compilation of state-by-state figures on student achievement. In that kind of area, where new indicators are needed but have to be established by the states, the C.C.S.S.O. center performs a function that the Center for Statistics and the federal government are not able to perform right now.
Q What is your major criticism of current national education data?
A The data simply do not provide a lot of information that we need about the educational system in this country. When the National Commission on Excellence in Education was doing its report, commission members were looking for indicators of the dimensions and quality of education in the nation.
One of the things that came out very clearly and very strongly is that for a lot of key concerns-levels of achievement, what kids are presented in school, the quality of teaching-the information simply is not there. It is not being collected.
Q Why doesn’t it exist?
A There are two or three reasons. One is that everybody would like to have the information but nobody is willing to pay for it in terms of the cost and effort that are required. A pretty good example here is information on the content of instruction in schools. Another example is information on achievement. We have ways of getting that information. They are relatively costly. And with a limited budget for the collection of education statistics in this country, there is only so much we can do.
Another reason is that there will have to be a process of technical development before we can collect a lot of the information that we would like to have. For example:
All of the data that we have on teaching right now is indirect. We have information on teachers’ S.A.T. scores when they are in high school. We have information on their grade-point average and how many of them were in the lower quartile of their class as high-school students and candidates for teacher education. We have absolutely no—and that is not an overstatement—direct information on the professional quality of the teaching workforce in this country. We have no idea if it is getting better or worse. We have no idea how teachers in this country compare in their ability to teach with teachers in other countries.
On the other hand, we are now, from research and from professional practice and experience, developing pretty clear notions of what constitutes high-quality teaching that could be operationalized. But there will have to be a process of turning those concepts into procedures and then taking on the effort and expense of collecting that information.
Q What about criticisms that federal data always appear too late to be useful?
A I agree that much of the federal information is not useful because it is too old. Again, that is partly a resource question and partly an issue of tradeoffs between having accurate data and having timely data. Currently, the situation is that you can have very accurate information from two school years’ back, because it has come in, it has been thoroughly vetted out and verified with the states, and then it has been published once we were absolutely sure that it was accurate. Meanwhile, the National Education Association asks the states to estimate where they stand now. Well, those estimates are just clearly projections. There is no assurance at all that they are really accurate.
Q How will the center address that problem?
A That is a major responsibility of the center. We have a project contract with the federal Center for Statistics to identify areas where timeliness seems to be a problem. Do the data sit around on desks at the Center for Statistics or do the requests sit around on desks at the state level? We have to figure out where the down time is, and then we are simply going to have to bring state people together and ask them, “How can this be improved?”
States get student-achievement information out very quickly. And if we can test kids allover a state and come up with school-building-level averages in three months—which most states are able to do—then what I envision doing is seeking the advice of state information-systems people and data specialists about how we could shorten the time that is required to produce some of the other information.
Q What is your opinion of the government’s ‘wall chart’?
A The wall chart is not useful. The outcome measures—especially the achievement measure that is used, the A.C.T. and S.A.T. scores—are so misleading that they are really unsatisfactory. That achievement variable, in my opinion, ought not to be in the chart. When you have got 6 percent of the kids in one state taking the S.A.T. and over 50 percent of the kids in another state taking the S.A.T., you have to ask, what subsection or what stratum of the student population do those groups represent?
The other problem I have with the wall chart is that the 50-state rankings make no attempt to group states by the kinds of constraints they face.
The major contribution that the chiefs’ program is making to this field of educational indicators is accounting for the different circumstances that states face-so that Iowa, for example, is compared with states that have similar resources, similar demographics, similar populations, and are trying to do similar things, and Connecticut is compared with states that are similar to Connecticut. That makes a hell of a lot more sense.
Q What do you think of the Center for Statistics’ indicators project?
A I, like many others, have at this point almost no idea what the project consists of. I do not think the Center for Statistics knows. The indicators project came out last year. I understand that there is some possibility that what were reported as indicators will become the core data reported in a regular publication of the Center for Statistics, such as The Condition of Education.
It is admirable that the Center for Statistics is interested in identifying a core set of important statistics that would be reported clearly to the policymakers and the public on a regular basis, because in the past the N.C.E.S. program and publications have not really met that need. There have been enough changes and eclecticism in the content of the publications so that you could not pick them up every year and say, “This is where we stand this year.”
The indicators represent a good effort to identify the 20 or so indicators that are available, that can be reported on, by the federal Education Department. But what are we doing to expand that list so that we go after those things that we do not know about? I would hate for that list, which has a lot of gap around it, to become the set of indicators that we report on m education for the next 25 years.
Q Is it even feasible to compare states?
A When we start clustering states, those clusters will be formed largely by the desires of the states. There is a certain amount of interest in regional clustering. Regional economies are fairly similar, and state may be more interested in comparing themselves with adjoining states than with states on the other side of the country. They are going to be interested in comparing themselves with others that are trying to do the same things.
Q Are the two techniques that the center is talking about using—namely, clustering like states and developing composite scores—going to mean that no 50-state rankings will be available?
A The council membership has talked a lot about this. Some people think that we just simply should not produce 50-state rankings. There is no way to prevent that. The N.E.A. already produces lots of rankings of the states on lots of variables. The College Board scores are available, and whatever we do, I assume the secretary of education is going to continue to put out a 50-state ranking of the College Board scores. There is really no way to head those off. On some variables, 50-state rankings are interesting.
But I find that on most individual variables—like looking at teacher salaries without accounting for the cost of living in the state—it does not tell you very much. Th me, a better comparison is when you cluster states into states with like circumstances or like goals or like program features or different program features and then do comparisons to see whether one state seems to be doing better than others.
Q How can you do student-achievement assessments without creating what is in effect a national curriculum?
A In the first place, in order to assess any subject, whether it is social studies or reading or math, the states have to get together and determine what they can agree on that they all pursue in those areas.
That is not identifying a national curriculum. It is establishing where states have enough common ground to be compared.
We may discover that the states are doing so much different that there is very little common ground, and the comparable student achievement that can be reported upon is a relatively narrow band of information. Along with establishing the areas of commonality, we also ought to allow for some flexibility, so that states that are doing other things that are supplemental or optional, in addition to the common core, can measure those things.
It is not the purpose of this program to comprehensively assess everything that the states are conceivably trying to teach.
[State officials meet next month under the C.C.S.S.O.'s auspices to begin collective planning on how to measure student achievement across states.]
Q Are you worried that the chiefs’ measure of student achievement will duplicate state testing programs or place an additional testing burden on states?
A That is a real important issue and it is almost a paramount concern on our part. In the first place, kids should not be burdened with a whole lot of testing. Second, states already have in place testing programs, sometimes multiple programs, to measure student achievement.
If they can have a limited sample of kids in the state participate in the state-to-state assessment program, and if they can link statistically the state-to-state assessment program with their own testing program so that if things stay the same, we can estimate their relative standing year to year after that based on how kids do in their own program, then we do not even have to interfere or disrupt.
Q Some chiefs have expressed concern about the costs of the indicators project. NAEP, in particular, has been associated with high costs. How do you respond to those concerns?
A There are several reasons why the NAEP program is relatively costly. They draw their own sample-a probability, representative sample of the state-and they do it with their own staff in a very statistically proper way, which is expensive. It is possible for states to do that kind of representative sampling in other ways using their own staff that would not incur that expense.
Q Do you see any other way of developing achievement data as being just as costly?
A It really is important to distinguish the fact that we are not recommending that states contract with NAEP to participate in the concurrent program. That has been presented to states as costing $150,000 to $250,000 a year, and that is not what we are recommending. We are recommending that the center coordinate an effort where states review and select items available from the national assessment that have been developed, that have been pilot-tested, and designed to measure student achievement, and then administer those items in their states in a coordinated program. The advantage to using the NAEP item pool is that it was nationally funded. It is in the public domain. And there are national normed data.
[The center has started a working group to coordinate planning with NAEP officials. The group, which includes representatives from the Center for Statistics and other state-based organizations, like the National Governors Association, met for the first time in January. According to Mr. Selden, NAEP officials have agreed to help in planning the chiefs’ collection of student-achievement data.]
Q Is the indicators project coming too late in the reform movement?
A No, I do not think it is too late at all. My impression is that legislators and governors are sophisticated enough to recognize that if they invested in education two years ago, there is a reasonable amount of time that has to transpire before a lot of the effects show up.
Second, a lot of states have the beginnings of sets of indicators in place, so some evidence will be forthcoming on the impact of those reforms even as our program is put in place. Also, the reforms were put in place because people believed they ought to work. If there is not an indicator now or next year to measure that, I think policymakers will be willing to go on faith for a while.
