Opinion
Federal Opinion

The What Works Clearinghouse: Time for a Fresh Start

By Robert E. Slavin — December 18, 2007 6 min read
  • Save to favorites
  • Print

The What Works Clearinghouse is the flagship initiative of the federal Institute of Education Sciences. Begun in 2002, it was intended, according to its Web site, to “help the education community locate and recognize credible and reliable evidence to make informed decisions,” and to provide educators with a “central and trusted source of scientific evidence of what works in education.” Scientifically valid and clearly written reviews of research on practical programs are essential to evidence-based reform, so the clearinghouse has been eagerly awaited by all who believe that educational practice should emphasize programs with strong evidence of effectiveness.

BRIC ARCHIVE

After five years and more than $30 million, the clearinghouse has finally begun to produce significant numbers of reports on the evidence base supporting various educational programs. But the reports make it clear that the clearinghouse has failed. Its arcane and poorly justified procedures have produced information that is neither scientifically justified nor useful to educators.

Recently, the What Works contract was awarded to a new contractor, Mathematica Policy Research Inc. This transfer provides an opportunity for a fresh start—one that is needed if the clearinghouse is to accomplish its worthy goals.

What is wrong with the What Works Clearinghouse is that although its rules appropriately emphasize random assignment, they ignore design elements with far more potential for bias than lack of random assignment. As a result, the clearinghouse gives its highest ratings for evidence of positive effects to programs supported by studies that are often very small, very brief, very biased, and/or very seriously flawed in other ways, failing to give educators valid or meaningful information on the programs they might use to improve their students’ achievement.

One example is in the middle school mathematics topic area. The clearinghouse gave its top rating, “positive effects,” to only one program, Saxon Math. Two randomized and four matched studies met clearinghouse standards.

See Also

How can the What Works Clearinghouse provide practioners with more useful information about education research? Join the discussion.

To get into the top category, a program must have significant effects in at least one randomized study and one other study. The unpublished randomized study that qualified Saxon Math for the “positive effects” rating involved 46 students taught by one teacher in one high school. The only outcome measure was made up by the author, and is closely aligned with the Saxon Math curriculum (but not the curriculum used in the control group). The other small randomized study found no differences, and two of the four studies that used conventional measures of math not keyed to the Saxon Math curriculum found effects favoring the control group. The median effect size across the four studies that used conventional measures was only +0.06 (most researchers consider an effect size, the proportion of a standard deviation separating experimental and control groups, to be educationally meaningful if it is +0.20 or larger).

Another egregious example relates to a program called DaisyQuest, computer software designed to teach phonemic awareness in grades K-1, listed in the clearinghouse as having “positive effects” on “alphabetics.” The DaisyQuest studies involved about five hours of computer instruction. Sample sizes were extremely small: 49 in one study, 27 in another, 69 in a third. Worse, outcome measures included activities taken from the DaisyQuest program (which experimental students had practiced and control students had never seen). In fact, control students were not being taught phonemic awareness at all. In one of the studies, ignored in the ratings, a comparison treatment was used in which a teacher taught phonemic awareness to a group of children, and those children scored far better on the Phonological Awareness Test than those who experienced DaisyQuest (effect size = -0.44).

Studies like those of DaisyQuest are the rule, not the exception, among programs rated “positive” in the clearinghouse’s beginning-reading topic report. Other programs rated “positive” included Kaplan SpellRead, a tutoring program validated as having “positive effects” in alphabetics by an eight-week study in a single school in Newfoundland involving 47 children. Another tutoring program, Stepping Stones to Literacy, was rated “positive” for alphabetics based on a five-week study involving just 36 children, in which tutoring was delivered by project staff members.

A four-week study of an ill-defined intervention called Peer Tutoring and Response Groups evaluated a process-writing model in which 4th graders in the experimental group worked in small groups to plan, draft, edit, and finalize compositions. The small groups were composed of English-language learners and fully proficient English-speakers. On the final test used as the outcome measure, children were asked to write a composition. In the experimental group, children were allowed to help each other write their compositions, while children in the control group wrote by themselves. Not surprisingly, experimental ELL students wrote significantly more words in their compositions, with the help of their English-proficient groupmates. This outcome qualified Peer Tutoring and Response Groups for a “positive effects” rating for “English-language development” in the English-language-learning topic report.

Program ratings should not be strongly influenced by one or two small studies, but should emphasize programs evaluated with many students in many schools.

The clearinghouse is unaccountably inconsistent from topic to topic. It requires that studies have a duration of at least a semester in math, but there is no duration requirement in reading or in programs for English-language learners. This means that if DaisyQuest, Stepping Stones to Literacy, SpellRead, or Peer Tutoring and Response Groups had been math programs, their key studies would have been excluded. The middle school math review considers only textbook programs, ignoring computer-assisted instruction and programs that focus on changing instructional processes (such as cooperative learning). Elementary math includes computer-assisted instruction, but not instructional-process programs. Other reviews of math programs have found that the instructional-process programs excluded by the clearinghouse have the strongest positive effects in the most rigorous evaluations, yet a reader of the What Works Clearinghouse would never know this.

The current clearinghouse will not pass muster among the scientific community, among educators, or among policymakers. Its Web site should be taken down immediately while new procedures are devised. The new procedures should refocus on giving educators unbiased information on the likely outcomes of programs available to them today to accomplish their most important goals, as outlined in the What Works Clearinghouse’s mission statement. To this end, the procedures should place strict standards on outcome measures (to remove those biased toward the experimental group) and on duration (requiring at least 12 weeks of intervention), to ensure that highlighted studies are meaningful and fair. To give appropriate emphasis to large, unbiased studies, the clearinghouse should compromise on statistical adjustments for clustering and other statistical issues that do not introduce bias. Program ratings should not be strongly influenced by one or two small studies, but should emphasize programs evaluated with many students in many schools.

Some day, the What Works Clearinghouse could become a reliable, respected source of information regularly consulted by educators and policymakers as they make critical decisions for children. As it currently stands, however, the clearinghouse is counterproductive, communicating to educators, policymakers, and researchers that research to date has little to offer them in choosing proven or promising programs. Mathematica has an opportunity to make a fresh start with a new clearinghouse that truly represents the accumulated findings of high-quality research.

Educators and policymakers have been promised fair, meaningful, and useful information they can rely on to make wise decisions for children. With a fresh start, the What Works Clearinghouse can still fulfill this promise.

Related Tags:

A version of this article appeared in the December 19, 2007 edition of Education Week as The What Works Clearinghouse: Time for a Fresh Start

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Academic Integrity in the Age of Artificial Intelligence
As AI writing tools rapidly evolve, learn how to set standards and expectations for your students on their use.
Content provided by Turnitin
Recruitment & Retention Live Online Discussion A Seat at the Table: Chronic Teacher Shortage: Where Do We Go From Here?  
Join Peter DeWitt, Michael Fullan, and guests for expert insights into finding solutions for the teacher shortage.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
The Science of Reading: Tools to Build Reading Proficiency
The Science of Reading has taken education by storm. Learn how Dr. Miranda Blount transformed literacy instruction in her state.
Content provided by hand2mind

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal What’s Behind the Push for a $60K Base Teacher Salary
When reintroduced in Congress, a bill to raise teacher salaries will include money to account for regional cost differences.
5 min read
Teachers from Seattle Public Schools picket outside Roosevelt High School on what was supposed to be the first day of classes, Wednesday, Sept. 7, 2022, in Seattle. The first day of classes at Seattle Public Schools was cancelled and teachers are on strike over issues that include pay, mental health support, and staffing ratios for special education and multilingual students.
Teachers from Seattle Public Schools picket outside Roosevelt High School on what was supposed to be the first day of classes, Wednesday, Sept. 7, 2022, in Seattle. The first day of classes at Seattle Public Schools was cancelled and teachers are on strike over issues that include pay, mental health support, and staffing ratios for special education and multilingual students.
Jason Redmond/AP
Federal Teachers Shouldn't Have to Drive Ubers on the Side, Education Secretary Says
In a speech on priorities for the year, U.S. Education Secretary Miguel Cardona said teachers should be paid competitive salaries.
5 min read
U.S. Secretary of Education Miguel Cardona delivers a speech during the “Raise the Bar: Lead the World” event in Washington, D.C., on Jan. 24, 2023.
U.S. Secretary of Education Miguel Cardona delivers a speech during the “Raise the Bar: Lead the World” event in Washington, D.C., on Jan. 24, 2023.
Sam Mallon/Education Week
Federal A Chaotic Start to a New Congress: What Educators Need to Know
A new slate of lawmakers will have the chance to influence federal education policy in the 118th Congress.
4 min read
Rep. Kevin McCarthy, R-Calif., talks on the House floor after the first vote for House Speaker when he did not receive enough votes to be elected during opening day of the 118th Congress at the U.S. Capitol, Tuesday, Jan 3, 2023, in Washington.
Rep. Kevin McCarthy, R-Calif., talks on the floor of the U.S. House of Representatives on Jan. 3 following the first round of voting for House Speaker. McCarthy fell short of enough votes to be elected speaker in three rounds of voting on opening day of the 118th Congress at the U.S. Capitol.
Andrew Harnik/AP
Federal Historic Changes to Title IX and School Safety Funding: How 2022 Shaped K-12 Policy
Federal lawmakers sought to make Title IX more inclusive, respond to school shootings, and crack down on corrupt charter schools.
6 min read
Revelers march down Fifth Avenue during the annual NYC Pride March, Sunday, June 26, 2022, in New York.
Revelers march down Fifth Avenue during New York City's annual Pride March in June. Proposed changes to Title IX would explicitly protect students from discrimination based on their gender identity or sexuality.
Mary Altaffer/AP