School & District Management

New Tool Alerts Teachers When Students Give Up on Tests

By Sarah D. Sparks — September 05, 2017 5 min read

How much can a test really tell you if a student gives up while taking it?

Quite a lot, as it turns out—if teachers know exactly when and how a student disengages. That’s what some schools nationwide started to do earlier this summer, using real-time alerts during a computer-based adaptive test to spot students going off task, and salvage meaning from assessments that students think they are writing off.

“It’s really a shift from seeing [testing] simply as a way to improve achievement scores—which it does—to seeing it as a source of data that teachers can use to understand their students,” said Jim Soland, a research scientist with the research and testing firm Northwest Evaluation Association.

The alerts are part of an ongoing research project between NWEA and the 50,000-student Santa Ana school district, a member of the California Office to Reform Education, or CORE, district consortium.

3Engage web

Teachers in the mostly poor and Hispanic Santa Ana district have always spotted the signs of students shutting down on a test, but they couldn’t tell how badly it affected their performance until the test was already complete.

“They play with the keyboard, they may be fidgeting,” said Emily Wolk, the assistant director of research and evaluation for Santa Ana, whohas watched over her share of tests. “Of course, we are very concerned about that; we know there is a connection between their engagement on a test and how they do in other areas as well.”

Gauging the Speed of Thought

Schools generally use two ways to measure student engagement on a test. Teachers can look at the results of the test and compare it to what they know of the student to decide how hard he worked, or they can ask a student directly if she gave the test her best effort. Both measures are subjective, and aside from noticing a student physically putting her head down, it can be difficult for teachers to realize a student has checked out during a test until it’s already been turned in. Even then, it’s hard to tell the difference between a student who is distracted because he had a fight with his mom over breakfast and one who despairs because a question is too hard.

Instead, NWEA researchers Stephen Wise, Soland, and Nate Jensen measure student-response times to hundreds of questions on the NWEA’s MAP Growth test, an adaptive, computer-based assessment taken by about 20 percent of U.S. students. It tracks the difficulty of the questions it presents to students’ previous test performance.

Wise and his colleagues found that under normal circumstances, students take 40 to 50 seconds to read and answer each question. If students starts to disengage from a test, “they start answering items very quickly—2 to 3 seconds—more quickly than it would take to evenread the question,” Wise said.

By tracking this “rapid guessing,” Wise and his colleagues can monitor how much effort students put into each question in real time and compare it with how hard students and teachers think the students worked. “People had thought that you are either engaged or disengaged on a test, and if you disengage, you stay disengaged,” Wise said, “but we found there’s no set pattern. It’s not like people shift into not trying and stay there. It’s more that they size up an item when they see it, and if it looks like more effort than I’m prepared to give, I’m just guessing.”

Who Guesses?

On average, the researchers found that after students first start rapid-guessing, they still legitimately try to answer 80 percent of the remaining questions on a test. Only 1 percent to 2 percent of those who disengage from a test rapid-guess on all or most of their questions.

Elementary students are less likely to disengage on a test than those in middle and high school—where as many as half of students rapid-guess at least once in a given test, Soland said. While there are relatively small racial differences in disengagement, boys are significantly more likely than girls to start rapid-guessing.

The subject matters, too. Though students often report math as moredifficult than reading, the researchers found that students on average are nearly twice as likely to rapid-guess on a reading test as on a math test, and they are more likely to guess at individual questions on any test that requires more reading.

Soland and Jensen now are working with Santa Ana to connect frequent test disengagement to other problems in school. They found that students who are less skilled at communicating with adults and classmates were more likely to disengage, as were those with more of a fixed mindset about academic skills—the belief that such skills are innate, rather than built with practice. But most of all, a student’s likelihood of disengaging on a test was associated with his or her self-management and self-regulation skills, the ability, for example, to show up for class prepared and on time. “As they disengage from tests and the course material, a whole host of other things come up ... attendance, suspensions, course failure ... that have been connected to risk of dropping out of school,” Soland said.

“What we’re really showing is lack of test engagement is a symptom around a lot of deep-rooted problems,” Soland said. “In my mind, there’s this chain from, if a kid has low motivation, a lack of self-belief in academic subjects, that can manifest itself in a lot of different ways.”

Soland and Jensen are now using the tests to build practical measuresof students’ social-emotional development and connection to school under the first national Social-Emotional Assessment Design Challenge award by the Collaborative for Academic, Social, and Emotional Learning.

NWEA is also changing its test reporting to show a student’s engagement levels as well as a performance score. It would show, based on the student’s performance before she started guessing, how much better she could have performed by trying harder.

“Eventually, people are going to start viewing test data differently: not just, here is your score, but here is how engaged people were when they gave you these scores; here’s how well does this reflect what students know and can do,” Wise said.

In the pilots so far, alerting the teacher to disengagement during the test “had a profound effect on [students’] engagement,” Wise said, though there are no formal evaluations of the intervention yet.

“This is a tool,” said Santa Ana’s Wolk. “A teacher knows the students, knows how to quietly go over and say, ‘Hey, how’s it going? I care about you, and I’m concerned you are moving too quickly here.’ ”

Coverage of learning mindsets and skills is supported in part by a grant from the Raikes Foundation, at www.raikesfoundation.org. Education Week retains sole editorial control over the content of this coverage.
Coverage of learning mindsets and skills is supported in part by a grant from the Raikes Foundation, at www.raikesfoundation.org. Education Week retains sole editorial control over the content of this coverage.
A version of this article appeared in the September 06, 2017 edition of Education Week as New Tool Alerts Teachers When Students Give Up on Tests

Events

Student Well-Being Webinar Boosting Teacher and Student Motivation During the Pandemic: What It Takes
Join Alyson Klein and her expert guests for practical tips and discussion on how to keep students and teachers motivated as the pandemic drags on.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
A Holistic Approach to Social-Emotional Learning
Register to learn about the components and benefits of holistically implemented SEL.
Content provided by Committee for Children
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
How Principals Can Support Student Well-Being During COVID
Join this webinar for tips on how to support and prioritize student health and well-being during COVID.
Content provided by Unruly Studios

EdWeek Top School Jobs

Interdisciplinary STEAM Specialist
Smyrna, Georgia
St. Benedict's Episcopal School
Interdisciplinary STEAM Specialist
Smyrna, Georgia
St. Benedict's Episcopal School
Arizona School Data Analyst - (AZVA)
Arizona, United States
K12 Inc.
Software Engineer
Portland, OR, US
Northwest Evaluation Association

Read Next

School & District Management New York City's Equity-Minded Schools Chief Resigns
Richard A. Carranza, the chancellor of the New York City schools, announced Feb. 26 he will step down from the job next month.
4 min read
Richard Carranza, Chancellor of the New York City Department of Education, arrives to Public School 188 The Island School as students arrive for in-person classes, on, Sept. 29, 2020, in the Manhattan borough of New York.
Richard A. Carranza announced he will depart the top New York City schools job in March.
John Minchillo/AP
School & District Management Opinion New Resource Tracks School System Reopening
The Return to Learn Tracker identifies the current instructional model of all regular public school districts with three or more schools.
5 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
School & District Management San Francisco School Board Pauses Renaming 44 Schools, Promises to Consult Historians
The renaming of 44 schools in the San Francisco Unified School District is apparently being put on hold after intense blowback.
Greg Keraghosian
1 min read
A pedestrian walks below a sign for Dianne Feinstein Elementary School in San Francisco, on Dec. 17, 2020. The San Francisco Unified School District put the renaming of 44 schools, including Dianne Feinstein Elementary School, on hold after local and national blowback.
A pedestrian walks below a sign for Dianne Feinstein Elementary School in San Francisco, on Dec. 17, 2020. The San Francisco Unified School District put the renaming of 44 schools, including Dianne Feinstein Elementary School, on hold after local and national blowback.<br/><br/>
Jeff Chiu/AP
School & District Management Superintendent Who Led During COVID-19 School Shutdowns Gets Top Honors
Michelle Reid of Washington state's Northshore district, one of the very first to close schools last March, was named National Superintendent of the Year.
3 min read
Michelle Reid, superintendent of the Northshore district in Washington
Michelle Reid, the superintendent of the Northshore district in Washington, was named National Superintendent of the Year.
courtesy of AASA, the School Superintendents Association