Opinion Blog

Rick Hess Straight Up

Education policy maven Rick Hess of the American Enterprise Institute think tank offers straight talk on matters of policy, politics, research, and reform. Read more from this blog.

Education Opinion

Evaluation: Just Like a Blender?

By Guest Blogger — May 28, 2014 6 min read
  • Save to favorites
  • Print

Note: Andrew McEachin, an assistant professor of educational policy analysis and program evaluation at NC State University, is guest posting this week.

I promised that I would spend today talking about promising teacher evaluation models, but I got sidetracked by a recent New York Times article on the Obama administration’s desire to create a college rating system. In the article, Jamienne Studley, a deputy secretary at the U.S. Department of Education, is quoted as saying rating the U.S. college system is like “rating a blender ... This is not so hard to get your mind around.” The comparison of rating a dynamic social system to a kitchen appliance is a gross oversimplification of how difficult it is to build a valid, reliable, and trustworthy public accountability model. It is also a stark reminder of the disconnect between what researchers know about the design and impact of accountability systems, and what is implemented by federal, state, and local policymakers. If creating sound policy was as easy as Studley claims, why do we consistently see policies adopted that fly in the face of decades of social science research? I will instead spend today discussing a few of the largest gaps between the research and policy community with respect to teacher and school accountability policies.

Don’t get me wrong, part of the problem is that the research community does not do a good enough job disseminating our work to the greater education stakeholder community (I applaud organizations like Scholars Strategy Network and Policy Analysis for California Education in their efforts to close this gap). With that said, Studley’s comment, and many of the accountability policies implemented to date, reminds me of paper by Professor Steven Kerr titled On the Folly of Rewarding A, While Hoping for B. The issues below are prime examples of hoping for one outcome, but due to design and implementation issues, measuring another.

The education production process is multi-dimensional

To be an effective educator, regardless of how “effective” is defined, teachers have to take on a number of different roles, each with its own requisite skills, knowledge, and duties. However, when it comes time to hold teachers accountable for their students’ outcomes, we typically only measure a few of the dimensions of teacher quality. As Kerr noted, it is silly to think that holding teachers accountable for a few objective measures of student performance will get them to improve in all areas of teacher quality. A recent paper by Kirabo Jackson showed that teachers have a profound effect on students’ non-cognitive outcomes, and these non-cognitive outcomes are important predictors of success in postsecondary education and the labor market. If we continue to narrowly define teacher quality, there is little reason to believe that teachers will improve along a number of important, but unmeasured, dimensions.

Parents and other stake-holders have different values and expectations for schools and student outcomes

Just as the education production process is multi-dimensional, so too is the list of desired knowledge, skills, and traits we expect students to acquire in our schools. In a seminal article Professor David Labaree outlines the competing, although not mutually exclusive, philosophies/goals that guide what we expect out of our schools. More recently, in a string of studies, colleagues from Michigan State have evaluated how parent and other stakeholder satisfaction is influenced by the design of an accountability model, and how the backgrounds of the stakeholders influence their schema of an ideal school (here and here). This is all to say that once you factor in the multi-dimensionality from the production side (teachers, principals, etc), and the consumer side (parents, the general public, etc), the process becomes much more difficult than whether my blender makes a delicious smoothie.

Proficiency rates are noisy and tightly linked to student demographics...yet we still use them

One of the main complaints with the NCLB-style accountability model is its reliance on holding schools accountable for students’ proficiency rates on math and reading assessments. These measures are strongly correlated with the types of students schools serve, predictably over-identifying schools with larger shares of minority or low-income students as failing. A second problem is the use of year-to-year changes in proficiency rates as a measure of school improvement. These year-to-year changes are rife with statistical noise, akin to flipping a coin to deciding a school’s fate. And yet, the ESEA waivers still require states to use proficiency rates as a measure of school success (here). In most states, even those that use growth measures, proficiency rates make up a preponderance of a school’s accountability grade, index, or other summative rating. The use of proficiency as rates the main criterion by which we define school success will continue to penalize the schools that need the most support.

Context is important, but we don’t want to do anything about it

It is a well-known fact that roughly three-quarters of a student’s math or reading test score is driven by factors that occur outside of a school. Therefore, if you want to get at the true impact of a teacher or school on a student’s outcomes, you need to account for these non-school related factors. The NCLB-style proficiency rate system did not account for these factors. The ESEA waiver program allows states to implement models based on changes, or growth, in student achievement in an effort to hone in on the true impact of teachers and schools on students. However, the type of growth models allowed by the feds does not account for a majority of the contextual differences among schools. In fact, the growth models (e.g., Student Growth Percentiles) are only marginally less correlated with student demographics than proficiency rates (see here and here). We therefore shouldn’t expect to see any major differences between the NCLB era and the ESEA waiver era in terms of which schools are labeled as failing (more on this on Friday).

More years of data are better

While you might be thinking that this is obvious, it doesn’t appear to be to policymakers. We have known for a long time that using multiple years of data to generate teacher and school performance measures is more reliable than using data from a single year (here and here). However, even with the opportunity to construct a new accountability system under the ESEA waiver process, most states are still relying on just one year of data to place schools in performance categories. Given that most states have collected at least school-level performance data for the better part of the past 10 years, it is reasonable to assume that states could readily implement longer periods of time in their accountability systems.

I think Studley’s analogy is more apropos of the current policy landscape: Let’s throw some ideas into a blender, mix some stuff around, and see what sticks. Instead of implementing more of the same, I am, perhaps naively, optimistic that there are opportunities for researchers, policymakers, practitioners, parents, and other stakeholders to come together to design an accountability model that is valid, fair, reliable, and trustworthy. The research community is just starting to understand how school-level accountability policies impact students, teachers, and so on, and now we’re talking about holding universities and teacher programs accountable. For all the issues listed above, there is a positive role that accountability policies can play in the future of U.S. public education. We just need to slow down, work together, and stop pretending this stuff is easy. Tomorrow I will discuss a few ideas for the design of teacher and school accountability that I find most compelling.

--Andrew McEachin

The opinions expressed in Rick Hess Straight Up are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class
Teaching Profession Live Online Discussion What Have We Learned From Teachers During the Pandemic?
University of California, Santa Cruz, researcher Lora Bartlett and her colleagues spent months studying how the pandemic affected classroom teachers. We will discuss the takeaways from her research not only for teachers, but also for

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Schools Get the Brunt of Latest COVID Wave in South Carolina
In the past few weeks, South Carolina has set records for COVID-19 hospitalizations and new cases have approached peak levels of last winter.
4 min read
Two Camden Elementary School students in masks listen as South Carolina Gov. Henry McMaster talks about steps the school is taking to fight COVID-19, Wednesday, Sept. 15, 2021, in Camden, S.C. McMaster has adamantly and repeatedly come out against requiring masks in schools even as the average number of daily COVID-19 cases in the state has risen since early June. (AP Photo/Jeffrey Collins)
Education More States Are Requiring Schools to Teach Native American History and Culture
Advocates say their efforts have gained some momentum with the nation’s reckoning over racial injustice since the killing of George Floyd.
3 min read
A dancer participates in an intertribal dance at Schemitzun on the Mashantucket Pequot Reservation in Mashantucket, Conn., Saturday, Aug. 28, 2021. Connecticut and a handful of other states have recently decided to mandate students be taught about Native American culture and history. (AP Photo/Jessica Hill)
Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP