School & District Management

‘Mischievous Responders’ May Skew Survey-Based Studies of Teens

By Holly Kurtz — May 15, 2014 4 min read
  • Save to favorites
  • Print

Anyone who has ever tried to survey high school students about anything has probably come across some responses like this:

Q: Three or four years ago, when you attended elementary school, did you get free or reduced-price meals at school?

A: I brought my lunch each day. School lunch was unrecognizable.

Or this:

Q: Last school year, did you ever have an unexcused absence or ditched class?

A: No, but why would I tell?

These nuggets of silly sarcasm have their upside, especially if they offer a much-needed moment of amusement during the otherwise tedious task of inputting hundreds of paper-and-pencil survey responses into a spreadsheet. But they can also have a dark side. According to an article published Thursday in the peer-reviewed journal Educational Researcher, so-called “mischievous responders” can pose a serious threat to the validity of survey-based research studies.

“Mischievous responders” are particularly problematic when researchers are trying to draw conclusions about categories of students who comprise a relatively small percentage of the overall population, suggests article author Joseph P. Robinson-Cimpian, an assistant professor of quantitative and evaluative research methodologies at the University of Illinois at Urbana-Champaign. That’s because, when the target group is small, it only takes a small handful of misleading results to dramatically skew the findings.

Robinson-Cimpian’s examples of skewed results often have ugly undertones. In short, the so-called “mischievous responders” appear to find it amusing to misidentify themselves as members of certain groups and then to lie about engaging in a host of undesirable, illegal or troubling behaviors.

For example, Robinson-Cimpian describes the case of a study that aimed to use a survey to compare adoptees with non-adoptees. When researchers followed up with in-person and/or parent interviews, they found that, of the 458 teens who had identified themselves as adoptees, 88 (or 19 percent) had lied. Because they had also exaggerated their low self-esteem, drinking habits, truancy, and physical problems, the researchers initially overestimated the risks that adopted children faced. Upon discovering their errors, they removed the “mischievous responders” from the data set and issued a retraction of their findings.

In another example provided by Robinson-Cimpian, researchers who followed up with in-person interviews found that all but two of the 253 teens who identified themselves as using artificial limbs had actually lied: Their limbs were intact.

Unfortunately, it is not always possible or even ethical to follow up on survey results with in-person or parent interviews. For instance, Robinson-Cimpian notes that researchers could put closeted LGBTQ students at risk by asking their parents to confirm their children’s reported sexual orientations. Yet “mischievous responders” seem to enjoy misidentifying themselves as LGBTQ students.

Robinson-Cimpian proposes using statistical methods to identify teenagers who provide extremely unusual patterns of survey responses.

“If we find that youths reporting to be gay are more likely than those reporting to be straight to say that they are blind and deaf and extremely tall and parenting multiple children all at the same time, then we might question whether the data are valid,” he said in a news release issued by the American Educational Research Association, which publishes Educational Researcher. “Just like these jokester youths think it’s funny to say that they are gay and blind, they also think it’s funny to say that they are suicidal, engage in sexually risky behavior, and take drugs. And this can dramatically affect our estimates of risk.”

Once researchers have identified likely “mischievous responders,” they can check and see how, if at all, their results change if they exclude them from their analyses or account for the possibility that they may be lying.

Based on a previous study by another researcher, Robinson-Cimpian estimates about 12 percent of teen survey respondents may be lying. However, teenagers lie on surveys for a variety of reasons so the percentage of “mischievous responders” may be smaller.

To me, a troubling implication of this article is that some teenagers seem to find it entertaining to lie about being members of certain groups and to then pretend that they have engaged in illegal or self-destructive behavior. But Robinson-Cimpian noted that it is unclear whether “mischievous responders” are purposely linking risky behavior with membership in these groups.

“I agree with you that it’s troubling that mischievous responders seem to be concentrated among historically disadvantaged groups,” he said. “The only problem is that we’re not entirely sure if kids are linking all of these things together when they respond to individual items on a survey. So, kids may not be thinking, ‘Ah ha, I’m going to make gay kids looks suicidal.’ But rather, they may be thinking that it’s funny to say they are gay, and then separately thinking that it’s funny to say they are suicidal. The end result is troubling because we, as researchers, link these two responses together when we run our statistical analyses; but I’m just saying that we don’t know if kids are doing this linking.”

Related Tags:

A version of this news article first appeared in the Inside School Research blog.


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Opinion Best Ways for Schools to Prepare for the Next Pandemic
Being better connected to families and the community and diversifying the education workforce are some of the ways to be ready.
14 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
School & District Management From Our Research Center Educators' Support for COVID-19 Vaccine Mandates Is Rising Dramatically
Nearly 60 percent of educators say students who are old enough to receive COVID vaccines should be required to get them to attend school.

4 min read
Mariah Vaughn, a 15-year-old Highland Park student, prepares to receive a COVID-19 vaccine during the vaccine clinic at Topeka High School on Monday, Aug. 9, 2021.
Mariah Vaughn, 15, a student at Highland Park High School in Topeka, Kan., prepares to receive a COVID-19 vaccine at her school in August.
Evert Nelson/The Topeka Capital-Journal via AP
School & District Management 10 Ways to Tackle Education's Urgent Challenges
As the school year gets underway, we ask hard questions about education’s biggest challenges and offer some solutions.
2 min read
Conceptual Image of schools preparing for the pandemic
Pep Montserrat for Education Week
School & District Management Reported Essay Principals Need Social-Emotional Support, Too
By overlooking the well-being of their school leaders, districts could limit how much their schools can flourish.
7 min read
Conceptual Illustration
Pep Montserrat for Education Week