Assessment Commentary

What NAEP Scores Aren’t Telling Us

By Ian Rowe — April 18, 2018 5 min read

For two decades, as part of repeated research studies, thousands of participants from diverse backgrounds have watched the same video of college students playing basketball in a circle. Participants are told to count how many times the students wearing white shirts pass the basketball. Stunningly, roughly half of the participants become so distracted trying to count the passes that they completely miss something extraordinary: a student dressed in gorilla suit who walks into the middle of the scene and thumps her chest before walking out of the frame nine seconds later.

In the world of neuroscience, this phenomenon of being oblivious to the obvious is called “inattentional blindness.” This occurs any time we as human beings fail to notice a fully visible but unexpected object because our attention was on another task, event, or activity.

Inattentional blindness is an important concept to keep in mind now that the 2017 National Assessment of Educational Progress results for reading and mathematics for 4th and 8th grades have been released.

As many feared, results were extremely disappointing across the board. Nevertheless, there are already reams of analysis of certain subgroups highlighting the stubborn achievement gaps within the mesmerizing categories of students’ race and family income. For example, despite the fact that only 37 percent of all 4th graders were at or above “proficient”—further evidence that poor reading performance crossed all racial boundaries—the dominant reaction to the scores continues to focus on the black-white achievement gap.

Enter family structure in a full-body gorilla suit, undetected amid the obsessive counting of other NAEP groupings."

What we haven’t seen are NAEP analyses of achievement differences by family structure, even though we know the stability of the family within which children are raised matters monumentally to their educational outcomes.

Take a recent study from the Institute for Women’s Policy Research: By 2012, the number of single mothers in college had doubled in just 12 years to nearly 2.1 million. Despite their best efforts to create a better future for themselves and their children, only 28 percent of single mothers who entered college between 2003 and 2009 earned a degree or certificate within six years—an outcome that adversely affects both mother and child. For college-going women without children, the completion rate was more than double.

And according to the Centers for Disease Control and Prevention, there were more than 1 million births to women aged 24 and younger in 2016. Seventy-one percent of these births were to unmarried women, and 42 percent were to women who already had at least one child. There are always exceptions, but the magnitude in the number of multiple, non-marital births to typically unprepared young women and men, and the decades-long rise in the rate of single-parenthood among young mothers, create a much greater risk of fragile families that correlate with child poverty, chronic student absenteeism, and the kind of toxic stress that begins in utero and impairs long-term academic advancement.

So what does this mean for NAEP? To be clear, NAEP data are an invaluable resource. The test remains the largest continuing and nationally representative assessment of what our nation’s students know and can do.

NAEP reports information on student demographics, including race, ethnicity, socioeconomic status, gender, disability, and limited English proficiency. But reporting by these measurement groupings—though critical—has the unintended effect of fueling the fixation on those categories as the primary bases for comparison. This is particularly true with race, where differences in outcomes (e.g., the Hispanic-white achievement gap) are perceived to be caused by race-related reasons (e.g., racism), which often elicit only race-related interventions.

It is a classic case of what gets measured gets managed. And what does not get measured gets ignored.

Enter family structure in a full-body gorilla suit, undetected amid the obsessive counting of other NAEP groupings.

As it turns out, NAEP already includes a question on many student questionnaires that gets at the issue of family structure by asking students to identify the adults who live in their home from a range of guardian options.

While self-reported, and thus susceptible to reliability concerns, this little-known data collection is a promising start. But given the lack of consensus that family structure actually matters to academic outcomes—or the fear of retribution of publicly stating that it does—it will take real courage from the National Center for Education Statistics to lift this data from obscurity.

Unlike gender or race, family structure is not a static measure that is easily discernible. Consider the powerful analysis done by Maria Cancian and Daniel R. Meyer in their study titled “Implications of Complex Families on Poverty and Child Support Policy.” After monitoring 7,169 first-born children of unmarried mothers in Wisconsin between 1997 and 2007, they found that 10-year-old children of unmarried parents lived in increasingly complex family structures, as both the biological father and mother had more children in and outside of their relationship. Families change. Nevertheless, I challenge the technical experts at NCES to incorporate an actual or proxy measure for family structure into NAEP student groupings.

One suggestion from noted psychologist and researcher Nicholas Zill is that parents of a representative subsample of NAEP-tested students be contacted to gather information on marital status and father or mother presence in the household. Whether by better leveraging the existing data collected on student questionnaires or by seeking out supplemental information in follow-up interviews, NAEP researchers would do our nation a great service if they developed a common metric for family structure and stability that could estimate the effects of these forces on educational progress.

Virtually 100 percent of the research subjects who failed to witness the “invisible gorilla” upon first viewing immediately saw it the second time around. They usually expressed disbelief that they initially missed it. Once confronted with the truth, the viewers could no longer ignore the reality right before their eyes.

Leaders in education reform and the researchers at NCES have both the capacity and the responsibility to elevate family structure as a critical prism through which we evaluate our country’s educational progress, on par with race, class, and other key groupings.

Education reformers and policymakers have already begun the handwringing about 2017 NAEP disparities between race, class, and gender. Yet the inattentional blindness ignores a force at least as fundamental to human development. 2020 could be an opportunity for a new baseline to measure progress by the critical indicator of family structure, which has been hidden in plain sight for decades.

If we really want to cure our blindness and understand why our children are not making the progress we seek, we must make this essential and predictive measure invisible no longer.

Related Tags:

Follow the Education Week Commentary section on Facebook and Twitter. Sign up to get the latest Education Week Commentaries in your email inbox.
A version of this article appeared in the April 25, 2018 edition of Education Week as What NAEP Scores Aren’t Telling Us


School & District Management Webinar What's Ahead for Hybrid Learning: Putting Best Practices in Motion
It’s safe to say hybrid learning—a mix of in-person and remote instruction that evolved quickly during the pandemic—is probably here to stay in K-12 education to some extent. That is the case even though increasing
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Mathematics Webinar
Building Equitable Systems: Moving Math From Gatekeeper to Opportunity Gateway
The importance of disrupting traditional American math practices and adopting high-quality math curriculum continues to be essential for changing the trajectory of historically under-resourced schools. Building systems around high-quality math curriculum also is necessary to
Content provided by Partnership for L.A. Schools
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Measuring & Supporting Student Well-Being: A Researcher and District Leader Roundtable
Students’ social-emotional well-being matters. The positive and negative emotions students feel are essential characteristics of their psychology, indicators of their well-being, and mediators of their success in school and life. Supportive relationships with peers, school
Content provided by Panorama Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Opinion AP Exams Can't Be Business as Usual This Year
The College Board seems unconcerned with the collateral damage of its pandemic approach, writes an assistant superintendent of curriculum and instruction.
Pete Bavis
5 min read
Illustration of large boat in turbulent waters with other smaller boats falling into the abyss.
iStock/Getty Images Plus
Assessment Federal Lawmakers Urge Miguel Cardona to Let States Cancel Tests, Highlighting Discord
A letter from Democratic members to the new education secretary calls for an end to the "flawed" system of annual standardized exams.
3 min read
Jamaal Bowman speaks to reporters after voting at a polling station inside Yonkers Middle/High School in Yonkers, N.Y. on June 23, 2020.
Jamaal Bowman speaks to reporters after voting at a polling station inside Yonkers Middle/High School in Yonkers, N.Y. on June 23, 2020.
John Minchillo/AP
Assessment How Two Years of Pandemic Disruption Could Shake Up the Debate Over Standardized Testing
Moves to opt out of state tests and change how they're given threaten to reignite fights over high-stakes assessments.
9 min read
Image of a student at a desk.
Assessment A Plan for Standardized Test Scores During the Pandemic Has Gotten States' Attention
A testing expert says his idea would provide helpful data with key context, but said other measures about student well-being are crucial.
7 min read
HS class 1257213326