Federal

‘What Works’ Rates Programs’ Effectiveness

By Debra Viadero — May 16, 2006 4 min read
  • Save to favorites
  • Print

The Department of Education, in a bid to make its $15 million What Works Clearinghouse Web site more useful to policymakers and practitioners, quietly unveiled a new face for the site this month.

Begun in 2002, the What Works project is aimed at vetting research on educational interventions and programs so that decisionmakers can make informed choices about “what works”—or is likely to work—in their own schools.

The initial reports, posted on the Web site since 2004, mostly gauged whether individual studies met the clearinghouse’s tough evidence standards. So few studies passed muster, though, that clearinghouse operators worried that practitioners would get discouraged from using it.

To address the problem, the revamped Web site includes new “intervention reports” that contain program descriptions, information on costs to implement the program, and simple, Consumer Reports-style ratings on program effectiveness in specific areas.

Gone Fishing?

To calculate the ratings, clearinghouse reviewers will also be allowed to weigh some smaller studies with strong positive effects as showing “potentially positive” evidence that a program works, rather then disregard the findings for failing to achieve statistical significance.

View new intervention reports on character education from the What Works Clearinghouse.

“This is an attempt to deal with what to do with this whole literature when there’s so little literature out there,” said Grover J. “Russ” Whitehurst, the director of the department’s Institute of Education Sciences, which oversees the clearinghouse.

Mr. Whitehurst discussed the changes during the May 8-9 meeting here of the National Board for Education Sciences, a 14-member group appointed by the president to advise the institute on setting priorities for its work. While board members applauded the changes, some independent observers offered a more skeptical assessment.

“The problem they’re facing is they don’t have anything to report, so they’re fishing for other results,” said Frederic A. “Fritz” Moser, a senior consultant to the Consortium for Policy Research in Education, a five-university research collaboration housed at the University of Pennsylvania in Philadelphia. “They’re caught in a dilemma they’ve created by stressing [methodological] rigor. Now they want to get back to relevance.”

However, Rebecca Herman, the What Works project director and a principal research scientist at the Washington-based American Institutes for Research, said the clearinghouse was not retreating from its commitment to setting high standards for educational research.

“We’re trying to help people focus on the body of evidence for an intervention and not just one study at a time,” she said.

Positive Thinking

The clearinghouse used the new format to review four character education programs: An Ethics Curriculum for Children, an elementary-level program developed by the Pittsburgh- based Heartwood Institute; Building Decision Skills, developed by the Institute for Global Ethics in Camden, Maine, and aimed at middle and high school students; Facing History and Ourselves of Brookline, Mass., which focuses on World War II and the Holocaust; and Lessons in Character, distributed by Young People’s Press Inc. of San Diego, for use in elementary and middle schools.

Consumer Reports

For the first time, the Department of Education’s What Works Clearinghouse has posted consumer-style ratings on the effectiveness of instructional programs. The ratings judge whether studies provide evidence that the program produces changes in three areas—students’ behavior, their knowledge or attitudes, and their academic achievement. The ratings were applied first to four character education programs.

The six ratings are:
● positive
● potentially positive
● mixed
● no discernible effects
● potentially negative
● negative

Program Ratings
Behavior Knowledge, Attitudes, and Values Academic Achievement
Building Decision Skills Not reported Potentially positive effects Not reported
An Ethics Curriculum for Children No discernible effects No discernible effects Not reported
Facing History and Ourselves No discernible effects No discernible effects Not reported
Lessons in Character No discernible effects No discernible effects Potentially positive effects

SOURCE: U.S. Department of Education

The resulting reviews, posted on the Web site two weeks ago, judge the four programs on how effectively they produce changes in three areas: students’ behavior; their knowledge, attitudes, and values; and their academic achievement.

No program was found to show “positive” effects, the group’s highest rating. But two programs—Building Decision Skills and Lessons in Character—got “potentially positive” ratings in some categories.

Mr. Whitehurst told the board on May 8 that the introduction of the “potentially positive” category also allows reviewers to weigh positive evidence from research that is well designed but “underpowered”—in other words, studies that may include too few students or classrooms to produce effects large enough to pass technical tests of statistical significance.

“Part of what’s going on here is that we are reanalyzing all the studies because many were misanalyzed, and the underpowered problem is greater when you do the analysis correctly than it appears in the literature itself,” Mr. Whitehurst said.

Now that the new ratings system is in place, project officials said, the clearinghouse will begin rolling out more reports. Currently, the Web site posts reviews in only two subjects areas: middle school mathematics and character education.

The clearinghouse redesign is part of a broader effort underway at the Institute of Education Sciences to make educational research more relevant to practitioners and policymakers.

Urban Research

Mr. Whitehurst also announced at the meeting that the institute had forged a formal agreement with the Council of the Great City Schools, a Washington-based group that represents the nation’s largest urban districts, to collaborate on research-oriented projects. Among the efforts on the two groups’ to-do list: forming an Urban Education Research Task Force, made up of researchers and practitioners, to advise the institute on the kinds of studies needed to improve learning in big-city schools.

“It’s easy to be relevant. It’s easy to be rigorous,” Mr. Whitehurst told the board. “It’s difficult to be both rigorous and relevant.”


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal New Federal Team to Work on Puerto Rico School Improvement, Oversight
The Puerto Rico Education Sustainability Team will focus on creating better learning environments and improving financial management.
3 min read
Secretary of Education Miguel Cardona visits the Emilio Delgado School in Corozal on June 30, 2021 during a visit to Puerto Rico.
Secretary of Education Miguel Cardona visits the Emilio Delgado School in Corozal on June 30, 2021 during a visit to Puerto Rico.
Teresa Canino Rivera/GDA via AP
Federal Pandemic Tests Limits of Cardona's Collaborative Approach as Education Secretary
He's sought the image of a veteran educator among former peers, but COVID has forced him to take a tough stance toward some state leaders.
10 min read
Education Secretary Miguel Cardona and Schools Chancellor Meisha Porter speak to Mia Arias, 10, during their visit to P.S. 5 Port Morris, a Bronx elementary school, Tuesday, Aug. 17, 2021 in New York.
U.S. Secretary of Education Miguel Cardona and Schools Chancellor Meisha Porter speak to Mia Arias, 10, during a visit to P.S. 5 Port Morris, a Bronx elementary school, last month.
Brittainy Newman/AP
Federal White House Launches Hispanic Education Initiative Led by Miguel Cardona
President Joe Biden said his administration intends to address the "systemic causes" of educational disparities faced by Hispanic students.
2 min read
Education Secretary Miguel Cardona writes down and draws positive affirmations on poster board with students during his visit to P.S. 5 Port Morris, a Bronx elementary school, Tuesday, Aug. 17, 2021 in New York.
U.S. Secretary of Education Miguel Cardona visits students in New York City at P.S. 5 Port Morris, a Bronx elementary school in the Bronx last month.
Brittainy Newman/AP
Federal Feds Add Florida to List of States Under Investigation Over Restrictions on Mask Mandates
The Education Department told the state Sept. 10 it will probe whether its mask rule is violating the rights of students with disabilities.
3 min read
Surrounded by lawmakers, Florida Gov.Ron DeSantis speaks at the end of a legislative session on April 30, 2021, in Tallahassee, Fla.
Surrounded by lawmakers, Florida Gov.Ron DeSantis speaks at the end of a legislative session on April 30, 2021, in Tallahassee, Fla.
Wilfredo Lee/AP