A group of California districts has been working for years on a local-level accountability system that uses a broad range of indicators to determine if a school is successful. Those districts, called the CORE districts, have gained perhaps the most attention for a plan to factor measures of students’ social-emotional skills and academic mindsets into accountability scores for their schools.
Now, as the nation ushers in a new era of accountability with President Obama’s signing of the Every Student Succeeds Act, or ESSA, the CORE districts plan to offer the surveys they use to measure those student traits to states and other school systems in hopes other educators can learn from their research.
“We’ve heard repeatedly that the same challenge that CORE districts have faced in finding useful, valid, reliable measures are faced by many districts, schools, and program providers across the country,” said Sara Bartolino Krachman, executive director of Transforming Education, the organization that helped design the measures. “It just seemed like, if everybody is sort of grappling with the same questions, we should be very transparent about where we are and what we’ve learned along the way.”
As I’ve written before, plans to measure student traits for accountability purposes have drawn concern from some researchers who’ve argued that they are prone to flaws and biases and that using them to judge schools’ progress could have unintended consequences.
But the CORE’s leadership and the researchers who helped design the student surveys it uses say a few years of piloting and testing its questions have led to strong, consistent measures and that others can surely learn from that work. CORE designed its system under a unique local-level waiver from the requirements of the now expired No Child Left Behind law, and it is set to begin incorporating its student-level social-emotional learning data into schools’ accountability scores next year.
Shift in School Accountability
And this all matters because a major shift in federal education policy means states will explore incorporating more factors beyond test scores into their accountability systems. President Obama signed an overdue overhaul of the Elementary and Secondary Education Act Thursday, replacing No Child Left Behind with ESSA.
Under the new law, states must incorporate at least one measurable non-academic indicator, like student engagement, school climate, or school safety into their accountability systems. States may turn to more objective data they already track, like suspension rates, for their “other indicator.” But the law also leaves the door open for more states to consider measures of student factors like self-control, resilience, and growth mindset into their school scales. (The CORE districts have done a little of both, designing a school index that incorporates factors like student traits, test scores, suspension rates, graduation rates, and the rate at which English-language learners are reclassified as fluent.)
“It’s been interesting to see how the language around accountability seems to be working toward the same kind of multi-metric system we’ve been working on,” said Noah Bookman, CORE’s chief accountability officer.
CORE will release its survey in early 2016, offering a resource to others “who are committed to assessing social-emotional skills systematically using common measures that are valid, reliable, and scalable,” Transforming Education said in a news release. The surveys, which measure self-management, growth mindset, social awareness, and self-efficacy, were designed in two phases: a 2014 pilot of a longer list of questions with about 10,000 students and a larger field test of a refined list of them with about 450,000 students in 2015.
“Field test analyses conducted by Professor Marty West of Harvard confirm that it is feasible to reliably assess student social-emotional competencies at scale through student self-report and teacher report surveys,” Transforming Education said in a news release. “The student self-report survey used took approximately 20 minutes to administer and was statistically significantly predictive of important student outcomes such as GPA, test scores, attendance, and suspensions.”
It’s worth noting that West has also published separate research with several other notable researchers about the potential for reference bias in social-emotional and non-cognitive measures. In layman’s terms, that means students may use different measuring sticks to assess their own strengths and weaknesses. Students at schools with strong social-emotional learning programs, for example, may have a better understanding of what concepts like self-control mean, and they may rate themselves lower than their peers at schools without such programs.
Testing the Social-Emotional Measures
But CORE leaders say they believe their measures are relatively reliable and that they used a number of methods to test them.
In early trials, for example, CORE used “anchoring vignettes,” which are intended to mitigate reference bias in student surveys, Krachman said. Those vignettes provide students with a snapshot of a fictitious student. For example: “Sally is a student who frequently forgets her homework at school and lashes out at other students.”
“The idea is to give this kind of representation of what it might look like to be low, in this case on the self-management scale,” Krachman said, adding that the surveys also included a similar vignette demonstrating strong self-management.
But CORE leaders found those vignettes weren’t necessary.
“We’re really not seeing the evidence of reference bias that we were concerned we might see or that we’ve seen in other contexts,” Krachman said.
What’s Next for CORE?
Now that No Child Left Behind has been left behind, the federal waiver the CORE districts have been operating under will eventually go away as well. Six districts operate under that waiver: Los Angeles, Long Beach, Fresno, San Francisco, Oakland, and Santa Ana. The Garden Grove, Sanger, and Sacramento districts are not part of the waiver, but they have helped design the accountability system and plan to use it to track their schools’ effectiveness.
CORE’s leaders say they plan to use their measurements in the future, whether or not they are tied to high-stakes federal accountability. And leaders hope their work will inspire accountability plans for California and other states.
The tension about responsibly measuring noncognitive and social-emotional traits isn’t likely to go away any time soon. I’m curious to see other researchers weigh in as CORE’s measures are distributed and considered by others.
Interested in seeing CORE’s measures? You can contact Transforming Education to get further information as it becomes available in early 2016.
Related reading on social-emotional learning and growth mindset:
- Study Measures Which Teaching Traits Boost Student Agency, Mindsets
- Walton Family Foundation Invests in Research on Measuring Grit, Character
- Carol Dweck Revisits the ‘Growth Mindset’
- ‘Nation’s Report Card’ to Gather Data on Grit, Mindset
- Urban Districts Embrace Social-Emotional Learning
- What Do Students Need to Succeed? Guide Helps Educators Navigate the Research
A version of this news article first appeared in the Rules for Engagement blog.