In his campaign for the GOP presidential nomination, Wisconsin Gov. Scott Walker credits his push to get rid of teacher protections with boosting his state’s 3rd grade reading scores.
Louisiana Gov. Bobby Jindal, another contender in the crowded Republican field, claims his championing of expanded school choice helped hike his state’s high school graduation rates.
And former Florida Gov. Jeb Bush says his work on school accountability has translated into big achievement gains, including for minority students, in the Sunshine State.
So how much weight should voters give such claims?
Maybe not much, educational experts say. It’s very tricky to draw a direct cause-and-effect relationship between a particular policy—for instance, a governor’s decision to increase education spending or add more charter schools—and an improvement in student achievement, researchers say.
“In most cases, it’s almost impossible” to truly credit a particular policy or policymaker with gains in student outcomes, said Laura Hamilton, the associate director of RAND Education, a research organization, in Santa Monica, Calif.
“It’s hard to disentangle [changes that resulted from policy shifts] from changes in the teacher labor market, changes in the economy, other changes in the state,” she said. “It’s almost never appropriate to make a causal conclusion about a particular policy initiative that’s implemented statewide.”
Research Challenge
The best way to study a policy’s impact, researchers say, is to do a randomized controlled trial using two similar groups of students: one that got access to better-funded schools, for instance, and another that didn’t.
“With randomized controlled trials you can, with some confidence, attribute [gains] to the intervention,” said Laura Desimone, a professor at the University of Pennsylvania’s Graduate School of Education in Philadelphia.
But that sort of rigorous research is nearly impossible when it comes to new state K-12 policies, experts caution. For one thing, it’s hard to imagine voters would want some children—but not others—to benefit from a policy change, new program, or funding increase.
What’s more, candidates in some cases may be jumping the gun by citing an impact from policy changes that have yet to really take root.
Take Walker’s claim that the 2011 changes to teacher policies in Wisconsin increased 3rd grade scores on state tests. Weakening protections could arguably lead to a new and better teaching force.
But even if researchers designed an intensive study on the effects of Walker’s teacher policies on student achievement, it could take years to deliver a final verdict on their efficacy. That’s partly because it would take a while to get a new crop of teachers in place and for them to begin having any impact on student achievement, said Morgan Polikoff, an assistant professor of education at the University of Southern California’s Rossier School of Education.
There simply hasn’t been enough time since the policy changes were enacted to link them to any hike in test scores, Polikoff said.
Plus, 3rd grade reading scores on state tests haven’t risen consistently throughout Walker’s time in office, though they did part of the time, making it complicated for the candidate to offer sweeping conclusions about the effects of his policies.
Walker’s campaign did not return calls to discuss the issue.
‘Lagging Indicators’
And then there’s Jindal’s contention that graduation rates have gone up in Louisiana, thanks in part to an expansion of school vouchers that he signed into law in 2012.
It’s true that the Pelican state saw a rise in graduation rates during Jindal’s tenure, from 71 percent in the 2010-11 school year to 74 percent in the 2012-13 school year, according to the Adjusted Cohort Graduation Rate, or ACGR, which schools have been required to use since 2008. (Note: Louisiana’s graduation rates also rose during Jindal’s tenure as calculated by the older, Averaged Freshman Graduate Rate metric, as did the nation’s.)
But graduation rates rose nationally too, from 79 percent to 81 percent during the same time frame on the ACGR. And some states saw much bigger jumps than Louisiana’s. Alabama’s graduation rate shot up 8 percentage points over the same period on that measure, for instance.
“If you’re seeing this nationally, it’s hard to argue that a particular state’s policy was responsible,” Hamilton said.
Plus, outcomes like graduation rates are often what’s known in the research world as a “lagging indicator"—it could take a long time for an intervention put in place with elementary school students to bear fruit farther down the line.
Stafford Palmieri, a senior policy advisor to Jindal, who came into office in 2008, stood by the governor’s claims that his education-redesign plans, including expanded school choice and what she described as greater educational accountability, have made a significant difference on graduation rates and other student outcomes. Saying otherwise “doesn’t pass the sniff test,” she said.
She acknowledged that there hasn’t been a robust research study of Jindal’s policies, but said that “common sense” dictates that his efforts are responsible for some of the gains the state has seen.
“We’ve made incredible changes in the system,” she said. “I don’t think graduation rates would have gone up if we’d done nothing.” And she added, “Each state has to decide for themselves what’s the right way to give every child an excellent education. In Louisiana, our reforms have given thousands of children the shot at a better education and a better life.”
Who Has Bragging Rights?
An apples-to-apples comparison of educational outcomes under the nine governors or former governors now running for president doesn’t show any standout stars, according to an analysis by the Education Week Research Center.
Data: The Governors Running for President and Their K-12 Records
Most of those candidates—whether Republican or Democratic—didn’t see huge spikes in selected math and reading scores on the National Assessment of Educational Progress, or NAEP, the nation’s report card, during their tenure in office. Neither did they see major improvements in graduation rates under the AFGR metric. (The analysis doesn’t include graduation rates before 2003.)
One potential exception may be Bush, who led Florida from 1999 to 2007. That period corresponded with gains in 4th grade reading on NAEP. Scores jumped from 206 on a 500-point scale in 1998, the year before Bush took office, to 226 in 2009, when he’d been out of office for two years. (Nationally, scores on the same exam rose from 213 to 220 during the same period.) And when it comes to closing the achievement gap, Florida made some substantial progress with respect to reading and math gaps in 4th grade, between 2003 and 2013. Nationally, the black-white gap in grade 4 reading closed by 4 points compared to 8 points in Florida. In grade 4 math, the nation narrowed the black-white gap by 1 point compared with 5 points in Florida.
But there’s no way to say for sure that Bush’s leadership—and the policies he put in place, such as ending social promotion for some students—were solely responsible for those gains, said Sterling Lloyd, the senior research analyst for the Education Week Research Center.
It’s possible that they made a difference, he said, but it’s important to take other factors—such as demographic changes—into consideration. And the jump for students overall wasn’t nearly as impressive for 8th grade reading.
On the Democratic side, Martin O’Malley, the former governor of Maryland, has cited the Old Line State’s number one overall grade for several years in Education Week‘s annual Quality Counts report as evidence of his strong stewardship of the state’s K-12 system. It’s true that Maryland was ranked number one for five years running, 2009 through 2013. But it’s hard to attribute that track record to any one policy, or policymaker, Lloyd said. And Education Week recently changed its methodology, pushing Maryland down to third place nationally in the 2015 report.
Typical Fare
A few headshaking education researchers aren’t likely to stop governors-turned-presidential-candidates from trumpeting student achievement gains that occurred under their watch, said Patrick McGuinn, an associate professor of political science at Drew University in Madison, N.J., who has studied the role education plays in elections.
For one thing, the strategy gives governors a chance to highlight their state executive experience—a quality that sets them apart from candidates who have served in, say, the U.S. Senate or the business community.
And a bump in test scores is also an easy-to-grasp sound bite that can help paint a candidate as a leader who gets results, McGuinn said.
“Voters like to hear hard facts. This is empirical, it’s data, it sounds great, it sounds positive, it sounds like a real result,” McGuinn said.
Governors have a myriad of student achievement results to choose from in making their case, everything from state test scores for a particular grade to Advanced Placement course-taking and college graduation rates. “They get to cherry pick whatever indicators and facts they get to use,” McGuinn said.
That’s partly why it makes sense to look beyond any single number—such as a rise in ACT scores or 4th grade math scores—when considering a candidate’s record, said Desimone of the University of Pennsylvania. She urged anyone wanting to understand these governors’ records to “look for the evidence across a host of different kinds of tests.” If scores have gone up across the board—and the gains have been sustained—that could bode well for a particular policy, she said.
