School & District Management

Ed. Dept. Quietly Funds More Experimental Studies

By Debra Viadero — December 11, 2002 6 min read
  • Save to favorites
  • Print

With the “No Child Left Behind” Act of 2001 emphasizing rigorous research, calls for more randomized trials in education studies have gained a new momentum in Washington policy circles.

Long before “rigorous research” and “randomized trials” became buzzwords in the field, however, the federal Department of Education quietly began setting aside more and more of its research dollars for experimental studies. According to the department, at least 16 such studies are in the works, nearing completion, or about to be funded over the next two fiscal years. That’s a big increase over previous years, the department’s research chief says.

“It’s very clear in No Child Left Behind that questions of what works in education will have high priority,” said Grover J. “Russ” Whitehurst, the director of the new Institute of Education Sciences, which oversees much of the research supported by the department. “Questions of what works link naturally to randomized trials.”

Common in research in medicine, pharmacology, and welfare reform, such studies entail randomly assigning subjects to either experimental or comparison groups. Such experiments are rarer and somewhat controversial in education.

For instance, of the 1,200 articles on mathematics and science education that were published between 1964 and 1998 in American Educational Research Journal, only 35 involved randomized trials, according to one study.

The apparent reluctance to use the methodology in education stems in part from concerns that such experiments can be expensive, unwieldy, and, in some cases, unethical. In addition, some experts contend, such studies often offer little help in understanding why an intervention works or doesn’t work.

“Good evidence is only as good as the theory that interprets it,” said E.D. Hirsch Jr., the University of Virginia professor who created the Core Knowledge school improvement program. Even though a well-respected, randomized study showed that reducing class sizes improved student achievement in Tennessee, he noted, efforts to do the same in California were not as successful.

“So what good did randomization do in this case?” Mr. Hirsch said.

Shrinking Pot?

The fear, other experts say, is that the Education Department’s new emphasis on randomized experimentation could shrink the pot of money available for studies using matched comparison groups, for descriptive studies, and for basic research.

“I don’t think that is likely,” said Mr. Whitehurst, who was the department’s assistant secretary for research and improvement before the creation of the new institute this fall. “By and large, these randomized trials will be funded out of evaluation and national- activities money, and that’s money that’s not previously been available to the research community.”

What’s more, he said, “we expect the pie to get larger.” He pointed out, for example, that President Bush’s Education Department budget request for fiscal 2003 called for increasing funds for core educational research by 44 percent, or roughly $46 million.

The experimental studies that the department expects to release soon, most of which were begun under President Clinton’s administration, gauge the effectiveness of the 21st Century Community Learning Centers after-school program, the Even Start family- literacy program, and Upward Bound, which prepares students to attend college.

Two of the ongoing, federally financed experiments include a study of the Success for All schoolwide improvement program, which is expected to cost $12 million over five years in public and private money, and a new, five-year, $30 million study examining the effectiveness of six different types of preschool programs.

In addition, the department is planning to spend $47 million over the next two fiscal years for randomized studies in 11 areas. The subjects are: early reading instruction, preschool literacy instruction, after-school programs, family literacy, alternative certification of teachers, professional development, educational technology, English-language learning, vocational education, charter schools, and adult literacy.

Mr. Whitehurst said department staff members analyzed the percentage of federally supported studies examining cause-and-effect kinds of questions that were addressed through experimental—though not necessarily randomized—methodology over the past two years. He said that percentage increased from 32 percent in fiscal 2001 to 100 percent in fiscal 2002.

Still, Mr. Whitehurst said, the department has no intention of counting out other kinds of studies.

Experts say randomized experiments are important because they are the soundest way available to find out what works. A not-yet-published study by three researchers at Mathematica Policy Research Inc., a Washington-based group, suggests, in fact, that investigators may get answers in randomized experiments that are different from those they get when they test the same questions using quasi-experimental methods, such as demographically matched comparison groups.

Some experts trace the movement at the federal level to conduct more randomized education studies to the 1998 passage of the Reading Excellence Act. That was the first call in federal education law for “scientifically based research.”

Clear Preference

The same terms and their definitions have since appeared in two other important federal education laws: the Education Sciences Reform Act, which revamped the Education Department’s research arm, and the No Child Left Behind Act. The latter, which is a reauthorization of the Elementary and Secondary Education Act, uses the term more than 100 times. (“Law Mandates Scientific Base for Research,” Jan. 30, 2002.)

While the definition does not exclude other types of education studies, it expresses a clear preference for randomized methodology.

“Getting the definition into the law became a catalyst for a much broader movement that was about to burst on the scene,” said Bob Sweet, who, as a staff member of the House Education and the Workforce Committee, helped draft that definition.

In some sense, the movement parallels one that took place in welfare reform in the 1970s and 1980s, according to Judith D. Gueron, the president of the Manpower Demonstration Research Corp., a New York City-based research organization. Since then, dozens of experiments have been conducted, leading to a major transformation in state and federal welfare programs.

“There were questions about the morality of doing this, questions about feasibility, and those questions exist in education,” Ms. Gueron said. “They have been overcome, but I think it is harder in schools. It’s going to be a harder approach that should be used carefully.”

The debate rages on in education. The National Research Council, in response to emerging concerns over what constitutes scientific research in education, published its own report on the subject last year.

This fall, in an effort to further that discussion among education researchers, the council convened a second group of prominent researchers. Among the topics on their agenda: randomized field trials and when it’s appropriate to use them.

“What we have now is an opportunity—and I’m hoping the education research community sees it as an opportunity—to respond to the call for better research and to think a little more rigorously about the relationship between the method and the question, and not to reject the idea of randomized trials,” said Michael J. Feuer, the executive director of the National Research Council’s division of behavioral and social sciences and education.

“In a way,” he said, “it’s a rather healthy development to see the policy community expressing a real strong interest in more and better education research.”

A version of this article appeared in the December 11, 2002 edition of Education Week as Ed. Dept. Quietly Funds More Experimental Studies


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Data Webinar
Working Smarter, Not Harder with Data
There is a new paradigm shift in K-12 education. Technology and data have leapt forward, advancing in ways that allow educators to better support students while also maximizing their most precious resource – time. The
Content provided by PowerSchool
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Deepen the Reach and Impact of Your Leadership
This webinar offers new and veteran leaders a unique opportunity to listen and interact with four of the most influential educational thinkers in North America. With their expert insights, you will learn the key elements
Content provided by Solution Tree
Science K-12 Essentials Forum Teaching Science Today: Challenges and Solutions
Join this event which will tackle handling controversy in the classroom, and making science education relevant for all students.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Democrats Try to Stamp Out School Closures as Omicron Surges
Democrats are speaking out more forcefully against COVID-19 school closures, recognizing a rising anger among parents.
6 min read
President Joe Biden greets Chicago Mayor Lori Lightfoot at O'Hare International Airport in Chicago, on Oct. 7, 2021. Democrats are speaking out against school closures even as the omicron surge puts additional pressures on public schools. Scattered teachers unions have called for closures, and a handful of districts have switched to virtual learning because too many educators have gotten sick.
President Joe Biden greets Chicago Mayor Lori Lightfoot at O'Hare International Airport in Chicago, on Oct. 7, 2021. Democrats are speaking out against school closures even as the omicron surge puts additional pressures on public schools. Scattered teachers unions have called for closures, and a handful of districts have switched to virtual learning because too many educators have gotten sick.
Susan Walsh/AP
School & District Management Explainer Teachers' COVID Sick Leave, Explained
Who gets paid? Are unvaccinated teachers eligible? How long should teachers be allowed to take off? Districts' answers will surprise you.
7 min read
sick leave 529156651 b
School & District Management Staff Shortages Are Bringing Schools to the Breaking Point
High rates of sick staff members have forced districts to move some of their schools temporarily back to remote learning.
6 min read
Image of staffing diagram.
Bill Oxford/E+
School & District Management Letter to the Editor The More Opportunities Students Get, the Better
Giving students jobs in schools shouldn't be controversial, writes U.S. Representative Virginia Foxx in this letter to the editor.
1 min read
Illustration of an open laptop receiving an email.