Anyone who has ever tried to survey high school students about anything has probably come across some responses like this:
Q: Three or four years ago, when you attended elementary school, did you get free or reduced-price meals at school?
A: I brought my lunch each day. School lunch was unrecognizable.
Q: Last school year, did you ever have an unexcused absence or ditched class?
A: No, but why would I tell?
These nuggets of silly sarcasm have their upside, especially if they offer a much-needed moment of amusement during the otherwise tedious task of inputting hundreds of paper-and-pencil survey responses into a spreadsheet. But they can also have a dark side. According to an article published Thursday in the peer-reviewed journal Educational Researcher, so-called “mischievous responders” can pose a serious threat to the validity of survey-based research studies.
“Mischievous responders” are particularly problematic when researchers are trying to draw conclusions about categories of students who comprise a relatively small percentage of the overall population, suggests article author Joseph P. Robinson-Cimpian, an assistant professor of quantitative and evaluative research methodologies at the University of Illinois at Urbana-Champaign. That’s because, when the target group is small, it only takes a small handful of misleading results to dramatically skew the findings.
Robinson-Cimpian’s examples of skewed results often have ugly undertones. In short, the so-called “mischievous responders” appear to find it amusing to misidentify themselves as members of certain groups and then to lie about engaging in a host of undesirable, illegal or troubling behaviors.
For example, Robinson-Cimpian describes the case of a study that aimed to use a survey to compare adoptees with non-adoptees. When researchers followed up with in-person and/or parent interviews, they found that, of the 458 teens who had identified themselves as adoptees, 88 (or 19 percent) had lied. Because they had also exaggerated their low self-esteem, drinking habits, truancy, and physical problems, the researchers initially overestimated the risks that adopted children faced. Upon discovering their errors, they removed the “mischievous responders” from the data set and issued a retraction of their findings.
In another example provided by Robinson-Cimpian, researchers who followed up with in-person interviews found that all but two of the 253 teens who identified themselves as using artificial limbs had actually lied: Their limbs were intact.
Unfortunately, it is not always possible or even ethical to follow up on survey results with in-person or parent interviews. For instance, Robinson-Cimpian notes that researchers could put closeted LGBTQ students at risk by asking their parents to confirm their children’s reported sexual orientations. Yet “mischievous responders” seem to enjoy misidentifying themselves as LGBTQ students.
Robinson-Cimpian proposes using statistical methods to identify teenagers who provide extremely unusual patterns of survey responses.
“If we find that youths reporting to be gay are more likely than those reporting to be straight to say that they are blind and deaf and extremely tall and parenting multiple children all at the same time, then we might question whether the data are valid,” he said in a news release issued by the American Educational Research Association, which publishes Educational Researcher. “Just like these jokester youths think it’s funny to say that they are gay and blind, they also think it’s funny to say that they are suicidal, engage in sexually risky behavior, and take drugs. And this can dramatically affect our estimates of risk.”
Once researchers have identified likely “mischievous responders,” they can check and see how, if at all, their results change if they exclude them from their analyses or account for the possibility that they may be lying.
Based on a previous study by another researcher, Robinson-Cimpian estimates about 12 percent of teen survey respondents may be lying. However, teenagers lie on surveys for a variety of reasons so the percentage of “mischievous responders” may be smaller.
To me, a troubling implication of this article is that some teenagers seem to find it entertaining to lie about being members of certain groups and to then pretend that they have engaged in illegal or self-destructive behavior. But Robinson-Cimpian noted that it is unclear whether “mischievous responders” are purposely linking risky behavior with membership in these groups.
“I agree with you that it’s troubling that mischievous responders seem to be concentrated among historically disadvantaged groups,” he said. “The only problem is that we’re not entirely sure if kids are linking all of these things together when they respond to individual items on a survey. So, kids may not be thinking, ‘Ah ha, I’m going to make gay kids looks suicidal.’ But rather, they may be thinking that it’s funny to say they are gay, and then separately thinking that it’s funny to say they are suicidal. The end result is troubling because we, as researchers, link these two responses together when we run our statistical analyses; but I’m just saying that we don’t know if kids are doing this linking.”
A version of this news article first appeared in the Inside School Research blog.