Federal

U.S. Review Finds No Proof That Reform Model Works

By Debra Viadero — February 01, 2008 4 min read
  • Save to favorites
  • Print

Despite attracting high-powered backers such as Bill Gates, the much-touted school improvement program known as First Things First has yet to muster conclusive scientific evidence to show that it prevents students from dropping out of school, a federal research review concludes.

The First Things First evaluation was one of two study reviews posted online Jan. 24 by the What Works Clearinghouse, which the U.S. Education Department set up to vet existing research evidence on the effectiveness of educational programs and practices.

Clearinghouse researchers gave a better rating to New Chance, the second dropout-prevention program that was reviewed.

They found “potentially positive” evidence showing that the now-defunct model improved high school completion rates for its target group of young welfare mothers.

The First Things First review drew criticism, though, from the program developers.

They contend that the federal reviewers set too high a bar for evidence and left out studies that gave the program high marks for improving student achievement, attendance, and graduation rates.

“I do not think it’s a fair and accurate representation of the evidence,” said James P. Connell, the architect of the program and the founder of the Institute for Research and Reform, the Toms River, N.J.-based group that houses it.

First Things First is now used in 30 secondary schools in seven states—California, Kansas, New Jersey, Mississippi, Pennsylvania, Texas, and Wisconsin—and more than two dozen elementary schools.

It is also on a short list of potentially successful high school redesign programs singled out by Microsoft Corp.’s chairman, Mr. Gates, as early as 2005.

The Seattle-based Bill & Melinda Gates Foundation has also spent millions of dollars helping schools implement First Things First.

Houston Evaluation

The high school-level program has three pillars: small learning academies that each keep students together from grades 9-12, a “family advocate” system pairing teachers with small groups of students for four years, and an emphasis on improving instruction. (“‘First Things First’ Shows Promising Results,” Mar. 9, 2005.)

Five studies have evaluated First Things First so far. In its analysis, though, the clearinghouse includes evidence from only one—an independent evaluation of three Houston high schools that implemented the program from 2001 to 2004.

Dropout Prevention

The What Works Clearinghouse dropout-prevention reviews focus on secondary school and community-based interventions designed to help students stay in school and/or complete school. These can include incentives, counseling, monitoring, school restructuring, curriculum design, literacy support, or community-based services to mitigate factors impeding progress in school.

BRIC ARCHIVE

SOURCE: Institute of Education Sciences

For that study, researchers compared academic outcomes for the 9th graders who entered the schools those years with those of their predecessors three years earlier. The study did the same calculations for 10 or 11 nonprogram schools and compared the results.

Based on that evidence, the federal reviewers concluded that the program showed “no discernible effects” on keeping students in school.

One problem with that conclusion, Mr. Connell said, is that “it’s limited to data from three schools in one city for one year—basically from freshman to sophomore year—and that’s really not a reasonable period of time for summarizing evidence.”

But Robert Wood, a senior economist at Mathematica Policy Research, the Princeton, N.J., group that runs the clearinghouse, said the review excluded evidence from the other four sites—Kansas City, Kan.; Riverview Gardens, Mo.; and Shaw and Greenville, Miss.—for methodological reasons.

For instance, in Kansas City, where the program was carried out districtwide, evaluators failed to find well-matched comparison schools.

The Mississippi study focused on measuring different outcomes, he said, and the Missouri evaluation only included one First Things First high school and one comparison school.

“You basically can’t separate the school from the program,”Mr.Wood added.

‘Hardest of Reforms’

The New Chance evaluation, likewise, rested on one study—a 10-state evaluation in which 2,000 women were chosen by lottery either to take part in the program or sit it out.

The participants took classes on parenting, life skills, and preparing for their high school equivalency exams.

When that phase ended, the women got assistance with job training and placement and access to child care.

The two new reports bring to 13 the number of dropout-prevention programs that the clearinghouse has evaluated so far.

Only two models—New Chance and Talent Search, a federal program that provides college and career counseling to academically talented youths from disadvantaged families—were deemed to have any evidence of success.

“High school reform is among the hardest of reforms to undertake,” noted Steven G. Seleznow, the program director for education at the Gates Foundation. Spurring change may be even harder for First Things First, he added, because the program works with existing school staff members rather than starting over with converted schools.

“In some places where we’re funding First Things First, we are seeing some evidence of success,” he said, “and in some places, we are not seeing evidence of success as fast as we would like.”

To get a better read on the program’s effectiveness, the foundation is supporting a comprehensive, independent evaluation of the program, not yet complete.

In addition, the federal Institute of Education Sciences this year launched a $6 million randomized study of First Things First that will involve 40 program sites around the country.

“I think I’m going to rely on the findings that will come from more thorough and focused research than a summary of other research projects,” Mr. Seleznow said.

A version of this article appeared in the February 06, 2008 edition of Education Week

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Budget & Finance Webinar
Innovative Funding Models: A Deep Dive into Public-Private Partnerships
Discover how innovative funding models drive educational projects forward. Join us for insights into effective PPP implementation.
Content provided by Follett Learning
Budget & Finance Webinar Staffing Schools After ESSER: What School and District Leaders Need to Know
Join our newsroom for insights on investing in critical student support positions as pandemic funds expire.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Achievement Webinar
How can districts build sustainable tutoring models before the money runs out?
District leaders, low on funds, must decide: broad support for all or deep interventions for few? Let's discuss maximizing tutoring resources.
Content provided by Varsity Tutors for Schools

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal A Bipartisan Bill Aims to Boost AI Education for K-12 Teachers
A new bill would create a grant program at the National Science Foundation focused on AI and K-12 schools.
4 min read
Highway directional sign for AI Artificial Intelligence
Matjaz Boncina/iStock/Getty
Federal K-12 Leaders Denounce Antisemitism But Reject That It's Rampant in Schools
Three school district leaders said they're committed to rooting out antisemitism during a hearing in Congress.
6 min read
From left, David Banks, chancellor of New York Public schools, speaks next to Karla Silvestre, President of the Montgomery Count (Md.) Board of Education, Emerson Sykes, Staff Attorney with the ACLU, and Enikia Ford Morthel, Superintendent of the Berkeley United School District, during a hearing on antisemitism in K-12 public schools, at the House Subcommittee on Early Childhood, Elementary, and Secondary Education, on May 8, 2024, on Capitol Hill in Washington.
From left, David Banks, chancellor of New York City schools, speaks next to Karla Silvestre, president of the Montgomery County, Md., school board; Emerson Sykes, staff attorney with the ACLU; and Enikia Ford Morthel, superintendent of the Berkeley Unified school district in Berkeley, Calif., during a hearing on antisemitism in K-12 public schools, at the House Subcommittee on Early Childhood, Elementary, and Secondary Education, on May 8, 2024, in Washington.
Jacquelyn Martin/AP
Federal Miguel Cardona in the Hot Seat: 4 Takeaways From a Contentious House Hearing
FAFSA, rising antisemitism, and Title IX dominated questioning at a U.S. House hearing with Education Secretary Miguel Cardona.
6 min read
Secretary of Education Miguel Cardona testifies during a House Committee on Education and Workforce hearing on Capitol Hill, Tuesday, May 7, 2024, in Washington.
U.S. Secretary of Education Miguel Cardona testifies during a House Committee on Education and Workforce hearing on Capitol Hill on May 7 in Washington.
Mariam Zuhaib/AP
Federal Arming Teachers Could Cause 'Accidents and More Tragedy,' Miguel Cardona Says
"This is not in my opinion a smart option,” the education secretary said at an EdWeek event.
4 min read
U.S. Secretary of Education Miguel Cardona speaks during Education Week’s 2024 Leadership Symposium at the Hyatt Regency Crystal City in Arlington, Va., on May 2, 2024.
U.S. Secretary of Education Miguel Cardona speaks during Education Week’s 2024 Leadership Symposium at the Hyatt Regency Crystal City in Arlington, Va., on May 2, 2024.
Sam Mallon/Education Week