School & District Management

Researchers: Measures of Traits Like ‘Grit’ Should Not Be Used for Accountability

By Evie Blad — May 13, 2015 6 min read
  • Save to favorites
  • Print

The debate over how to refer to so-called “noncognitive” student traits like self-control, grit, and gratitude is crowding out a more important conversation about how those traits should be measured and how to responsibly use the resulting data, two pioneering researchers in the field say.

Angela Duckworth, a University of Pennsylvania professor known for her research on grit, and David Yeager, a University of Texas researcher who has done extensive work in the field of growth mindsets, detailed their concerns in an essay in published today in Educational Researcher.

Because there are potential flaws in every existing measurement tool used to track the effectiveness of a growing number of efforts to build non-cognitive traits—known by names like social-emotional learning, character development, and 21st-century skills—such evaluations should not be used for accountability measures like school-to-school comparisons or teacher evaluations, the essay says.

“We share this more expansive view of student competence and well-being, but we also believe that enthusiasm for these factors should be tempered with appreciation for the many limitations of currently available measures,” it says.

When tracking interventions designed to boost desirable nonacademic traits and skills in students, “perfectly unbiased, unfakeable, and error-free measures are an ideal, not a reality,” the essay concludes. “Instead, researchers and practitioners have at their disposal an array of measures that have distinct advantages and limitations.”

There’s considerable debate over what to call these traits, qualities, and skills. In their essay, Duckworth and Yeager use the term “personal qualities” as shorthand for the rather wordy “positive personal qualities other than cognitive ability that lead to student success.”

As a growing body of research demonstrates how such qualities can relate to student success both inside and outside the classroom, educators and policymakers are increasingly emphasizing them in schools. Varying approaches emphasize different combinations of a menu of personal and interpersonal traits like persistence, self-control, self-awareness, resilience, empathy, gratitide, and curiosity.

But with the drive to change educational practices comes a desire to measure the effects of that change and the almost inevitable drive for accountability, and that’s where things get problematic, Duckworth and Yeager say.

Methods of Measurement

The essay uses the example of measuring self-control to explore the advantages and weaknesses of three possible forms of measurement—self-report questionnaires, teacher-report questionnaires, and performance tasks.

Questionnaires of both teachers and students are perhaps the most popular existing method for measuring growth of these personal qualities in students. In recent years, districts around the country have begun tracking student responses to questions about school climate, peer behavior, and personal attitudes, often cross-referencing the results with achievement data to look for trends.

Such questionnaires have advantages: They are inexpensive and relatively easy to develop and administer.

But they also have limitations that aren’t always acknowledged, Duckworth and Yeager write. The researchers say common problems like the following may lead to inaccurate results:


  • Survey participants may interpret questions differently than researchers intended.
  • Students or teachers may be limited in their ability to answer questions about growth in internal traits, such as motivation.
  • Surveys may fail to detect smaller incremental changes.
  • Reference bias—the comparative examples respondents use to gauge personal growth or success in some areas—may lead to different results from similar respondents.
  • “Social desirability bias” may lead students to give answers they think teachers want to see to certain questions rather than answering honestly.
  • Teachers may make assumptions about students when evaluating their behavior. For example, if they’ve determined that Sally is a “good kid,” then they might also determine that she has high levels of self-control, even if she hasn’t demonstrated it.

Performance tasks, another way of measuring personal traits, answer some of those concerns by using consistent, carefully crafted experiments to measure students’ responses and changes in those responses over time. In the case of self-control, a common performance task would be infamous “marshmallow test,” in which students are allowed to choose between eating a small pile of marshmallows now or an even bigger pile of marshmallows if they wait for a period fo time. The theory is that the students who delay gratification have more self-control.

But while the tasks themselves may seem like objective measures, the conclusions researchers draw from those tasks are often subjective, Duckworth and Yeager write. For example: “Is a child who refrains from playing with toys when instructed to do so exerting autonomous self-control, or does such behavior represent compliance with adult authority?”

Some other limitations the authors found with performance tasks include the following:


  • Created tasks may not reflect everyday life. In the case of the marshmallow test, a student seeking to resist temptation at home may cover up a bag of marshmallows with a cloth or walk away for a while to avoid eating them, which are reasonable self-control strategies, the essay says.
  • Other competencies, like hand-eye coordination, may affect a student’s performance on an unrelated task.
  • The results of certain tasks may be a less accurate over time as students grow accustomed to them, making it difficult to measure and track growth of certain traits.
  • Students may make random errors, like circling incorrect answers, when completing tasks.

So What?

Duckworth and Yeager aren’t suggesting that researchers and educators abandon efforts to measure students’ social, emotional, and character-based traits. Rather, they should exercise caution should be used in selecting a form of measurement and recognizing its limitations, the researchers write.

Mostly importantly, because of those limitations, special caution should be exercised in determining how the resulting data is used, the essay says. The problems posed by reference bias in questionnaires, for example, mean that their results should not be used to compare schools or classrooms for accountability purposes, the authors write.

“Current data and theory suggest schools that promote personal qualities most ably—and raise the standards by which students and teachers at that school make comparative judgments—may show the lowest scores and be punished, whereas schools that are least effective may receive the highest scores and be rewarded for ineffectiveness,” the essay says.

In other words, students in schools that do a good job emphasizing these skills may hold themselves to a higher standard, giving themselves lower scores on self-questionnaires.

Also problematic is the practice of using measurements of noncognitive traits to diagnose or assess the needs of individual students, it says. That’s because existing measures aren’t nuanced enough to measure progress on an individual level.

“Without highly reliable, multimethod, multiinformant measurement batteries whose validity has been demonstrated for diagnosis, it will be difficult for a practitioner to justify the individual diagnosis of children’s personal qualities, such as self-control, grit, or growth mind-set,” the essay says.

The authors also urge refinement of measurement methods so that educators can gather and track more reliable data on programs designed to build these traits, regardless of what schools choose to call them.

“Given the advantages, limitations, and medium-term potential of such measures, our hope is that the broader educational community proceeds forward with both alacrity and caution, and with equal parts optimism and humility,” Duckworth and Yeager wrote.


Related:

New Character Report Cards Rate Students on ‘Grit’
California’s CORE Districts Have Committed to Including Social-Emotional Factors in Accountability System

Follow @evieblad on Twitter or subscribe to Rules for Engagement to get blog posts delivered directly to your inbox.

A version of this news article first appeared in the Rules for Engagement blog.


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class
Teaching Profession Live Online Discussion What Have We Learned From Teachers During the Pandemic?
University of California, Santa Cruz, researcher Lora Bartlett and her colleagues spent months studying how the pandemic affected classroom teachers. We will discuss the takeaways from her research not only for teachers, but also for

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Some Teachers Won't Get Vaccinated, Even With a Mandate. What Should Schools Do About It?
Vaccine requirements for teachers are gaining traction, but the logistics of upholding them are complicated.
9 min read
Illustration of a vaccine, medical equipment, a clock and a calendar with a date marked in red.
iStock/Getty
School & District Management A Vaccine for Kids Is Coming. 6 Tips for Administering the Shot in Your School
Start planning now, get help, and build enthusiasm. It's harder than it looks.
11 min read
Cole Rodriguez, a 15-year-old student at Topeka West, gets a COVID-19 vaccine Monday, Aug. 9, 2021 at Topeka High School's vaccine clinic.
Cole Rodriguez, a 15-year-old student, gets a COVID-19 vaccine at Topeka High School's vaccine clinic.
Evert Nelson/The Topeka Capital-Journal via AP
School & District Management Letter to the Editor School Mask Mandates: Pandemic, ‘Panicdemic,’ or Personal?
"A pandemic is based on facts. A 'panicdemic' is based on fears. Today, we have both," writes a professor.
1 min read
School & District Management How 'Vaccine Discrimination' Laws Make It Harder for Schools to Limit COVID Spread
In Montana and Ohio, the unvaccinated are a protected class, making it tough to track and contain outbreaks, school leaders say.
4 min read
Principal and District Superintendent Bonnie Lower takes the temperature of a student at Willow Creek School as the school reopened, Thursday, May 7, 2020, in Willow Creek, Mont.
Bonnie Lower, a principal and district superintendent in Willow Creek, Mont., checks the temperature of a student as Willow Creek School reopened for in-person instruction in the spring.
Ryan Berry/Bozeman Daily Chronicle via AP