Opinion Blog

Rick Hess Straight Up

Education policy maven Rick Hess of the American Enterprise Institute think tank offers straight talk on matters of policy, politics, research, and reform. Read more from this blog.

Education Opinion

Three Great ‘Sleeper’ Studies to Check Out on Google Scholar

By Guest Blogger — July 24, 2017 4 min read
  • Save to favorites
  • Print

This month, Rick is out catching up on various and sundry projects that piled up during the rollout of Letters to a Young Education Reformer. In his stead, we’ve got a terrific slate of guest bloggers. Up this week is Collin Hitt, assistant professor of medical education at Southern Illinois University School of Medicine.

I used to collect records. Finding something good that was new to me, that was the magic. Still is. Then comes the fun of telling everybody about it.

Back in the days when I actually bought music, it didn’t matter to me if it was rare. It didn’t matter if it was on the rack at Best Buy. If it was interesting and new to me, that’s all that mattered. And if it was something I was sure my friends hadn’t heard, I ranted about it, with the naivest hope that they’d love it and buy it too. I actually considered buying every copy of “dwightyoakamacoustic.net” from the seven-dollar bin and handing it out for free.

Spotify changed the game. My rare records aren’t rare anymore (record-store slobs despair). Music is essentially free. It’s awesome. So now my friends have no excuse.

Neither do you, when it comes to important education research. Google Scholar has done for written research what Spotify has done for music. Great work is hiding in plain view, free for anyone to read—and I’m going to spin a few of my favorite sleeper studies this week.

Today’s episode: career academies, vocational schools, and career and technical education (CTE).

No need to re-frame this issue or to get hung up on definitions. This is a blog post not a policy brief. You’ve heard of vocational schools and the like. The question is, have you heard of these studies? Did they change your thoughts on the topic? They changed mine.

First up: a random-assignment study from Philadelphia, led by a former director of the federal Institute of Education Sciences, Ruth Curran Neild. Gold-standard design, which followed nearly 20,000 students from the time they entered enrollment lotteries through college graduation, and according to Google Scholar, it’s been cited only 13 times. I’d call that a sleeper.

A little background: Philadelphia embraced public choice during the Paul Vallas era, and a number of the city’s high schools focused directly on career and technical education. Perhaps my favorite line from the paper: “It is important to be clear that, in general, the CTE programs that Philadelphia’s students experienced did not garner any award or notice for being exemplary, nor were they cutting edge models of excellence.” The impacts of those unexceptional CTE schools:

CTE schools had higher on-time graduation rates in each of the three cohorts. This CTE advantage continued to five-year graduation rates for the two cohorts for which these data were available and to the six-year graduation rates for the one cohort for which data were available (13-27% increase in individual's odds of graduating using ITT estimate and 111-183% increase using Dosage estimate).

Those are large impacts. Then this:

Across the cohorts, CTE schools had virtually no impact on achievement growth from 8th to 11th grade. The CTE effect for learning growth in mathematics and reading comprehension was generally statistically insignificant, and the effects were always small.

The evaluation team followed up, once students entered college:

CTE schools had a positive impact on postsecondary enrollment, although the effects were not consistent across cohorts. For the Class of 2003, there were positive ITT, LATE, and Dosage impacts of CTE schools on enrollment in two-year colleges. For the Classes of 2004 and 2005, there were positive impacts on enrollment in four-year colleges. For the Classes of 2003 and 2004, there were positive impacts of enrolling in either a two-year or a four-year college.

For several reasons, they didn’t pool estimates across cohorts. If they had, the headline would be much simpler: significant and important gains in college enrollment.

Outside of large urban districts, regional vocational schools provide full-time CTE options. Massachusetts is a prime example, where Shaun M. Dougherty has done the best work. He uses an instrumental-variables approach to examine the impacts of these schools. And lest you doubt his sincerity in his belief of whether this approach provides plausible claims about cause and effect, see the title of his 2016 article, “The Effect of Career and Technical Education on Human Capital Accumulation: Causal Evidence from Massachusetts.” From the abstract:

All estimates suggest that participating in a high-quality CTE program boosts the probability of on-time graduation from high school by seven to ten percentage-points for higher income students and suggestively larger effects for their lower income peers, and for students on the margin of being admitted to oversubscribed schools.

Again, sizeable effects. In a secondary analysis he shows that students enrolled at the regional vocational schools outperform otherwise similar students traditional high schools who are taking heavy course loads of in-house CTE offerings.

Dougherty’s earlier work on this project was released by Fordham and got some notice, but the more developed version that appeared in Education Finance and Policy last year only has two citations so far. Come on people.

Career academies are another heavy duty CTE option. MDRC, with James Kemple playing lead, examined a number of 1990’s era career academies. Think of these as schools within a school. This was a nationwide study, with students admitted by lottery—so, experimental design. The final evaluation was released in 2008 and got far more notice than any other paper here, and that’s because it did what 99% of evaluations fail to do: it followed students into the workforce. No significant impact on test scores, on high-school graduation, on college going, or on college graduation. Yet, students who won a lottery to attend a career academy had significantly higher earnings by their mid-20s. Now that’s a research team that showed some grit.

The MDRC research has been the final word on career academies for almost a decade. There’s finally a new, rigorous study out from CALDER, led by Stephen Hemelt. His two co-authors are on staff at Wake County Public Schools, where the study is set. Suffice it to say, it’s rare to have district staff authoring gold-standard evaluations—so, good for them. From the abstract:

We find that enrollment in this academy increases the likelihood of high school graduation and college enrollment each by about 8 percentage points, with the attainment gains concentrated among male students. We also find that academy participation reduces 9th grade absences but has little influence on academic performance, AP course-taking, or AP exam success during high school.

No impact on test scores, again.

Not long ago I would’ve told you that the high-school evidence on career and technical education was thin. Now, I think it is one of the richest areas of school-choice research. And yes, whether you like it or not, vocational high schools are a form of school choice. The findings here are similar to what we see with voucher programs in New York, Milwaukee, and DC. Unimpressive impacts on test scores, but positive impacts on high-school graduation, college attendance, college graduation, and/or earnings.

Before it’s too late, spare yourself your usual school-choice rant. The findings from the CTE papers reviewed above have next to nothing to do with teachers unions.

So please, do a little reading. Do a little thinking. Or at least fire up “dwightyoakamacoustic.net.”

—Collin Hitt

The opinions expressed in Rick Hess Straight Up are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.