The White House and the Coalition for Evidence-Based Policy in Washington are doubling down on the use of experimental studies to find what works in policy and practice. The coalition awarded a pilot competition funding three low-cost randomized trials in education, health, and workplace safety. The three projects awarded grants this week will also take part in a conference with the White House Office of Science and Technology Policy in Washington on July 28.
Randomized control trials, held up as the gold standard for education research in the last decade of federal education policy, have also taken a lot of flak for being expensive, time-intensive, and likely to produce results so specific that they leave educators with not much more direction than they had before.
“We’re hoping to demonstrate a new paradigm—that RCTs, which have been seen as too expensive and too burdensome to be done on a regular basis, can, particularly with the use of administrative data, be much less expensive and be done with very little burden,” Baron said. “They are not just measuring short-term or intermediate outcomes. ... What we are looking for are demonstrations of interventions that have significant impacts on people’s lives.”
The Bottom Line
The education-related project will provide $159,000 to evaluate the long-term effects of Bottom Line, an intensive mentoring program for first-generation, college-going students in Massachusetts, New York, and Illinois.
The 18-year-old program has a long waiting list in each city for its services. Poor and low-income (defined as no more than 200 percent above the poverty line) high school juniors with at least a 2.3 grade-point average can apply to receive one-on-one support including matching their interests and skills with potential four-year colleges, applying for admission and financial aid, and enrolling. Students who attend one of the 36 colleges linked to the program also receive ongoing course-planning, career counseling, internship placement and other help for up to six years on the road to college graduation.
The program already has some evidence of effectiveness. Compared to the national average of about one in three college freshmen who ultimately graduate, 45 percent of Bottom Line students complete a degree in four years, and after six years, 84 percent of students graduate. That gave primary investigator Benjamin L. Castleman, an assistant education professor at the University of Virginia’s Curry School of Education, an opening to create a randomized trial without much disruption to the program, by changing the admissions process from first-come, first-served to a lottery system.
“The challenge is, it is not always easy to think about evaluating your program or setting it up in a way to allow for that sort of evaluation,” said Greg Johnson, the chief executive officer of Bottom Line, “but speaking from decades of trying to make the school board or foundation understand that your program makes a difference, having really hard solid evidence ... makes that much easier.”
The Bottom Line study will take seven years to complete, but, Baron said, “Sometimes you have to wait. If you have a program like Bottom Line that’s trying to not just get students into college but through college, the only way to shorten the timeline is to bend the space-time continuum.” But, he added, the study will ultimately show clear, long-term results of the program, which could be used to guide policy.
If successful, the Bottom Line study could serve as a model for easy evaluation of any program established enough to have a waiting list, Castleman said. “This competition is bringing a spotlight to 1) how fairly straightforward it is to change an application to allow this sort of rigorous evidence, and 2) that in the long run the kind of evidence this will generate can have a meaningful impact on the design of policy programs,” he said.
The other grants are:
- Health: A $183,000 evaluation of a nurse home-visiting program called Durham Connects will use postnatal hospital records to gauge whether the program reduces the use of emergency rooms by young mothers and babies..
- Workplace Safety: A $150,000 Occupational Safety and Health Administration study will look at whether businesses randomly chosen out of a pool of 29,000 for a safety inspection later have lower injury rates as well as lower sales or higher rates of business closures.