Opinion
Education Opinion

Gold-Standard Program Evaluations, on a Shoestring Budget

By Robert E. Slavin — October 05, 2011 3 min read
  • Save to favorites
  • Print

Note: This is a guest post by Jon Baron, President of the Coalition for Evidence-Based Policy, and Chairman of the National Board for Education Sciences

In today’s tough economic climate, quality evaluations of education reforms - to determine which are truly effective in improving student achievement, graduation rates, and other key outcomes - are especially important. They enable us to focus our limited resources on strategies that have been proven to work.

Well-conducted randomized controlled trials are generally recognized as the most reliable method (the “gold standard”) for evaluating a program’s effectiveness. However, widespread misconceptions about what such studies involve - including their cost - have often limited their use by education officials.

In plain language: Randomized controlled trials in education are studies that randomly assign a sample of students, teachers, or schools to a group that participates in the program (“the program group”) or to a group that does not (“the control group”). With a sufficiently large sample, this process helps ensure that the two groups are equivalent, so that any difference in their outcomes over time - such as student achievement - can be attributed to the program, and not to other factors.

Such studies are often perceived as being too costly and administratively burdensome to be practical in most educational settings. In fact, however, it is often possible to conduct such a study at low cost and burden if the study can measure outcomes using state test scores or other administrative data that are already collected for other purposes. Costs are reduced by eliminating what is typically the study’s most labor-intensive and costly component: locating the individual sample members at various points in time after program completion, and administering tests or interviews to obtain their outcome data. In some cases, the only remaining cost is the researcher’s time to analyze the data.

For example, the following are two recent randomized trials that were conducted at low cost, yet produced findings of policy and practical importance:

Roland Fryer, recent winner of the MacArthur “Genius” Award, conducted an evaluation of New York City’s $75 million Teacher Incentive Program in which 396 of the city’s lowest-performing public schools were randomly assigned to an incentive group, which could receive an annual bonus of up to $3000 per teacher if the school increased student achievement and other key outcomes, or a control group. Three years after random assignment, the study found that the incentives had no effect on student achievement, attendance, graduation rates, behavior, GPA, or other outcomes. Based in part on these results, the city recently ended the program, freeing up resources for other efforts to improve student outcomes.

The study’s cost: Approximately $50,000. The low cost was possible because the study measured all outcomes using state test scores and other administrative records already collected for other purposes.

Eric Bettinger and Rachel Baker conducted an evaluation of InsideTrack college coaching - a widely-implemented mentoring program for college students designed to prevent them from dropping out of school. This was a well-conducted trial, which randomized more than 13,000 students at eight colleges. The study found that the program produced a 14 percent increase in college persistence for at least two years, and a 13 percent increase in likelihood of graduating college.

The study’s cost: Less than $20,000. The low cost was possible because the study measured its key outcomes using administrative data that the colleges already collected for other purposes - i.e., their enrollment and graduation records - rather than by collecting new data through individual surveys.

In recent years, federal and state policy, as well as improvements in information technology, have greatly increased the availability of high-quality administrative data on student achievement and other key educational outcomes. Thus, it has become more feasible than ever before to conduct gold-standard randomized evaluations on a shoestring budget. Equipped with reliable evidence, education officials can have much greater confidence that their spending decisions will produce important improvements in student outcomes.

-Jon Baron

The Coalition for Evidence-Based Policy is a nonprofit, nonpartisan organization whose mission is to increase government effectiveness through the use of rigorous evidence about “what works.”

The opinions expressed in Sputnik are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Events

Jobs October 2021 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Data Webinar
Using Integrated Analytics To Uncover Student Needs
Overwhelmed by data? Learn how an integrated approach to data analytics can help.

Content provided by Instructure
Professional Development Online Summit What's Next for Professional Development: An Overview for Principals
Join fellow educators and administrators in this discussion on professional development for principals and administrators.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Vulnerable Students Left Behind as Schooling Disruptions Continue
The effects of unpredictable stretches at home can mirror those of chronic absenteeism and lead to long-term harm to learning.
4 min read
Students board a school bus on New York's Upper West Side on Sept. 13, 2021. Even as most students return to learning in the classroom this school year, disruptions to in-person learning, from missing one day because of a late school bus to an entire two weeks at home due to quarantine, remain inevitable as families and educators navigate the ongoing pandemic.
Students board a school bus on New York's Upper West Side on Sept. 13, 2021. Even as most students return to learning in the classroom this school year, disruptions to in-person learning, from missing one day because of a late school bus to an entire two weeks at home due to quarantine, remain inevitable as families and educators navigate the ongoing pandemic.
Richard Drew/AP
Education 'Widespread' Racial Harassment Found at Utah School District
The federal probe found hundreds of documented uses of the N-word and other racial epithets, and harsher discipline for students of color.
1 min read
A CNG, compressed natural gas, school bus is shown at the Utah State Capitol, Monday, March 4, 2013, in Salt Lake City. After a winter with back-to back episodes of severe pollution in northern Utah, lawmakers and Utah Gov. Gary Herbert will discuss clean air legislation and call for government and businesses to convert to clean fuel vehicles.
Federal civil rights investigators found widespread racial harassment of Black and Asian American students in the Davis school district north of Salt Lake City, Utah.
Rick Bowmer/AP Photo
Education Tiny Wrists in Cuffs: How Police Use Force Against Children
An investigation finds children as young as 6 and a disproportionate amount of Black children have been handled forcibly by police officers.
15 min read
Jhaimarion, 10, reacts as he listens to his mother, Krystal Archie talking with an Associated Press reporter in Chicago on Sept. 23, 2021. Archie’s three children were present when police, on two occasions, just 11 weeks apart, kicked open her front door and tore through their home searching for drug suspects. She’d never heard of the people they were hunting. Her oldest child, Savannah was 14 at the time; her youngest, Jhaimarion, was seven. They were ordered to get down on the floor.
Jhaimarion, 10, reacts as he listens to his mother, Krystal Archie talking with an Associated Press reporter in Chicago on Sept. 23, 2021. Archie’s three children were present when police, on two occasions, just 11 weeks apart, kicked open her front door and tore through their home searching for drug suspects. She’d never heard of the people they were hunting. Her oldest child, Savannah was 14 at the time; her youngest, Jhaimarion, was seven. They were ordered to get down on the floor.
Nam Y. Huh/AP
Education Gunman in 2018 Parkland School Massacre Pleads Guilty
A jury will decide whether Nikolas Cruz will be executed for one of the nation’s deadliest school shootings.
3 min read
Annika Dworet and her husband, Mitch Dworet, wipe away tears as their son's name is read aloud during Marjory Stoneman Douglas High School shooter Nikolas Cruz's guilty plea on all 17 counts of premeditated murder and 17 counts of attempted murder in the 2018 shootings, at the Broward County Courthouse in Fort Lauderdale, Fla. on Wednesday, Oct. 20, 2021. The Dworet's son, Nicholas Dworet, 17, was killed in the massacre.
Annika Dworet and her husband, Mitch Dworet, wipe away tears as their son's name is read aloud during Marjory Stoneman Douglas High School shooter Nikolas Cruz's guilty plea on all 17 counts of premeditated murder and 17 counts of attempted murder in the 2018 shootings, at the Broward County Courthouse in Fort Lauderdale, Fla. on Wednesday, Oct. 20, 2021. The Dworet's son, Nicholas Dworet, 17, was killed in the massacre.
Amy Beth Bennett/South Florida Sun Sentinel via AP