Fellows Program Links Researchers, Agencies
WASHINGTON--TO Floraline I. Stevens, the director of the program-evaluation and assessment branch of the Los Angeles Unified School District, one of the most important benefits her time at the National Center for Education Statistics has given her is peace and quiet.
"I read an article all the way through, without a phone call or a disruption," she observes, with some amazement. "That's not the life of a director of research and evaluation."
Ms. Stevens is spending the 1991-92 school year at the N.C.E.S. studying ways to develop measures of students' opportunities to learn classroom content, in order to improve the understanding of test-score data.
She is here as part of an unusual partnership between the federal agency and the American Educational Research Association that is aimed at bringing researchers and huge government data sets closer together.
In the process, the arrangement has benefited both the statistics center and the researchers.
Emerson J. Elliott, the acting commissioner of the N.C.E.S., says that, thanks to Ms. Stevens, the agency has become more aware of how to present information in ways that would be useful to educators.
"We're frequently frustrated that we don't produce things of value," he says. "People like Flo can think through how we do things."
For her part, Ms. Stevens says, she has been able to work on a research project that is of great interest to her, while gaining a better understanding of how the federal government works.
"I feel I have a found a network with the N.C.E.S. data-gatherers," she says. "I know who is responsible. I'm on a first-name, pick-up-the-phone basis [with them]."
An Investment in Infrastructure
The project that brought Ms. Stevens to Washington for a year came about following a series of discussions between Mr. Elliott and Richard J. Shavelson, then the president of the research association.
Mr. Elliott says that he was seeking a way to open up to researchers the vast storehouse of information that the federal education-statistics center collects.
Recalling an earlier partnership with the American Statistical Association, in which the N.C.E.S. brought leading statisticians to Washington for a year to learn about the "real world" of data collection, Mr. Elliott says he was "interested in a counterpart that would be less statistical, and more on the research side."
"They could study the content of education," he says. "That would benefit the center and themselves."
At the same time, Mr. Shavelson, the dean of the school of education at the University of California at Santa Barbara, was seeking a way to open new avenues for federal support of education research, which he notes had dwindled during the 1980's.
"My concern was that, at the time, the federal government wasn't investing in the infrastructure of education research," he says.
Mr. Shavelson also approached the National Science Foundation, which expressed an interest in the idea. The N.S.F. and the statistics center, an annex of the Education Department, joined together to provide a $500,000 grant to the A.E.R.A. to support a four-part research and training program.
The components include:
- A small-grant program, which provides three to five one-year grants for research on education policy and practice, particularly on mathematics and science education, using federal data sets.
To date, researchers supported under this program have studied such topics as why students drop out of middle schools, the making of minority scientists and engineers, and teachers' workplace conditions and their relation to retention.
- An institute on statistical analysis for education policy, held just before the A.E.R.A.'S annual meeting.
- A research-fellows program, which provides opportunities for young scholars-those working on or just completing their doctoral dissertations--to study policy-related research at one of the two agencies.
- A senior-fellows program, which allows prominent scholars to spend a year at one of the agencies to serve as a resource and conduct research on a topic related to the agency's mission.
The research association has been discussing with the agencies ways to expand and improve the program, Mr. Shavelson says. Currently pending is a grant proposal to extend the program for another three years.
In that proposal, he says, the statistics center and the association are looking at expanding the statistical-analysis training.
"Three days at a statistics institute only scratches the surface," Mr. Shavelson says.
In addition, he notes, the N.S.F. has asked for a greater focus on evaluation. He says the program would provide a series of training grants to universities to develop talent in that field.
The science foundation is also considering the creation of short-term "think tanks" that would bring together researchers to "chew on" a particular problem for six weeks in the summer.
While all the efforts so far have been highly regarded, Mr. Shavelson says, the senior-fellows program is perhaps the most prestigious.
To select senior fellows, he says, the federal agency discusses with the association the topic it wants help for, and the association identifies a leading scholar in the field.
Researchers at the senior-scholar level are usually invited to apply for fellowships, Mr. Shavelson explains.
"By the time you get to that level, if you don't get a phone call, you don't apply," he says. "People at that level aren't looking for work."
The first senior fellow at the N.S.F. IS Michael Huberman, a professor of education at the University of Geneva, who is developing a plan for disseminating research findings, as well as holding workshops on dissemination for staff members of the foundation.
At the N.C.E.S., the first senior fellow was James W. Guthrie, a professor of education at the University of California at Berkeley and a co-director of Policy Analysis for California Education, or PACE, a highly regarded research consortium in that state.
During the 1990-91 academic year, Mr. Guthrie helped develop a strategic plan for the statistics center at a time when a number of reports, such as that of the Special Study Panel on Education Indicators, proposed substantial changes in federal collection of education data.
"He helped us think through what we would do with the round of recommendations given to us," says Mr. Elliott.
In the second year, Mr. Shavelson says, the N.C .E .S. was looking for a "consumer" of data who could help the agency improve its reporting capabilities.
The association selected Ms. Stevens, the Los Angeles schools' research chief.
She also has experience with national issues as a member of the study panel on indicators and the Board on International Comparative Studies in Education, a National Academy of Sciences panel that oversees U.S. participation in cross-national studies.
Ms. Stevens says her selection represents a long-overdue recognition of the fact that not all research is conducted at a university.
"What it has done is brought legitimacy to the notion that school researchers do have qualifications and knowledge, and can work at a government agency to provide information," she says. "Normally, we don't get tapped for those things."
Bringing the 'Real World'
For her research project, Ms. Stevens chose to work on an issue that had occupied her attention in Los Angeles, as well as on the National Academy beard: developing a way to determine whether students have had the "opportunity to learn" the material on which they are tested.
"How can students do well on any assessment," she says, "if they had not been exposed to the content the assessment is developed for?"
Such a measure, she says, would indicate not only whether the material had been covered in the classroom, but also how well teachers had provided instruction.
The measures would be particularly important for low-income and minority students, she says, who often have the least skilled teachers.
In setting about determining how to construct such a measure, Ms. Stevens says, she began by surveying large districts to find out if they collect such data.
The survey found that "local school districts seem not to have looked at this as an important element in relation to student academic achievement at this point in time," Ms. Stevens says.
"All school districts have this capacity," she says. "Very few do it."
But her work has put the issue on the N.C.E.S. agenda, Mr. Elliott says. In fact, the agency and the Council of Chief State School Officers have scheduled a conference on the topic for next week.
"Flo influenced our thinking," Mr. Elliott says. "She made sense, and made more clear this is a direction we ought to be moving. It was not one we were initially enthusiastic about."
Ms. Stevens has also helped the agency by bringing to the table the perspective of a consumer of data, the acting commissioner says.
As an example, Ms. Stevens notes that she has pressed the N.c.E.s. researchers to define in their reports technical language that policymakers may not understand.
Without such definitions, she points out, district research directors may be reluctant to present information from the reports to school-board members who need it.
"My whole input has been how to make documents user-friendly so they are utilized by school districts," she says.
She also points out that she has served as a liaison between N.C.E.S. staff members and researchers and practitioners. "I have brought the real world to the N.C.E.S.," MS. Stevens says.