Is there a way to consistently and accurately measure students’ ability to do things like recognize emotions, solve peer conflicts, and exercise self-control? What if they don’t even realize those things are being measured?
The winners of a recent design challenge took a crack at it, proposing everything from analyzing students’ subtle behaviors when taking computer-based tests to asking them to play video games that simulate real-world experiences and gauging how they respond.
The contest winners responded to a prompt from a working group that is seeking to advance conversations about measurement. That group includes the Collaborative for Academic, Social, and Emotional Learning, or CASEL, researchers from several major universities, and the CORE districts, a group of large California school systems that have piloted social-emotional learning measures.
Social-emotional learning emphasizes the personal and relational skills researchers and employers say students need to succeed both inside and outside the classroom, skills like social awareness and self-management. Measurements that track how students are performing in these areas would help schools decide if their approaches and programs are successful, educators say, and some policymakers have pushed for more precise measurements that might allow for the inclusion of so-called non-cognitive skills in states’ school accountability models.
The Challenge of Measuring SEL
But designing those measurements is easier said than done. Schools commonly use self-reported surveys, through which they ask students to report on their own social-emotional learning progress or to assess the progress of their peers. But some researchers have warned that the results of such surveys shouldn’t be used for high-stakes purposes, like school funding or determining a student’s assignment to special interventions. That’s because students may respond to the same question differently depending on their own experiences, personal values, understanding of the question, and familiarity with the subject.
Some researchers think the future of SEL measurement lies in performance tasks, which track students’ behavior and responses to tasks, rather than their answers to questions. For most people, the most familiar example of a performance task is the “marshmallow test,” a social experiment through which researchers measured children’s self control by giving them a choice: They could eat one marshmallow right away or they could wait and eat more marshmallows later.
The first-place winner of the recent design challenge—an assessment organization called NWEA and the Santa Ana, Calif., school district—proposed a more subtle way to measure self-control that, unfortunately, doesn’t involve any candy. Instead, they propose analyzing metadata from computerized tests that show how quickly students respond to questions. NWEA researchers found that “rapid guessing,” or responding to a question so quickly that the student likely didn’t understand it, correlates with less engagement on tests and lower levels of personal skills like self-regulation.
Panorama Education, an organization that is best known for school surveys, got second place in the contest with “Social Detective,” which measures students’ ability to understand the thoughts and emotions of others by watching videos of strangers answering questions like “what makes a good friend?” The students are then asked about the people in the videos with questions like, “In making life decisions, what does Byron rely on most, reason or emotion?”
“The structure of the assessment is simple, yet accurately recreates an authentic context in which students need to exercise social perspective-taking skills,” Panorama expains in a demonstration on its website.
Check out some of the other noted social-emotional learning measurement methods on the results page.
Concerns About Performance Tasks
The design challenge winners raise some interesting ideas about what measurement should look like, but they don’t answer every concern raised by researchers. For one, each focuses on a narrow skill, rather than all five of the competencies that are typically emphasized in social-emotional learning programs.
And it’s possible for some of these measures to be gamed. With the winning proposal, for example, teachers pushing for better self-regulation scores could simply encourage students to stop and take a second before answering a question, even if they don’t know the answer.
The same researchers who sounded the alarm about measuring social-emotional learning with surveys also issued some cautions about performance-based tasks in a widely read 2015 paper.
While the tasks themselves may seem like objective measures, the conclusions researchers draw from them are often subjective, and engineered tasks may not reflect how students would respond in real-life situations, said Angela Duckworth, a University of Pennsylvania associate professor of psychology known for her research on grit, and David Yeager, an assistant professor of developmental psychology at the University of Texas at Austin who focuses on growth mindsets.
A student who chooses to quickly eat the marshmallow during the marshmallow test might not necessarily have low self-control, the paper said. Instead, the task might not reflect real life because it gives students limited options in how to respond: stare at the marshmallow or eat it. In real life, a person may exercise the self-control to resist eating the sweet treat by covering it up or by walking into another room to put it out of their mind, Duckworth and Yeager wrote.
Other things, like hand-eye coordination, could affect a student’s task performance, and they may grow more familiar with tasks if they are asked to repeat them over time, which may make the results less reliable, they wrote.
Bonus! Check out this video from the Education Week archives of a New York City teacher explaining how he uses the marshmallow test to explain self-control to his students.
Image: Getty