Up to this point, Data Detectives has focused on claims which lack quantitative backing due to simple errors in calculation. These are the most common types of errors you’ll see: people simply saying things without the requisite backing or people making some sort of statistical mistake that spirals. Today, however, we’ll be looking at a more minor mistake, one that anyone could make. What happens when you learn a statistic secondhand and try to extrapolate from there?
THE CLAIM
I was startled when, in the course of routine Googling, I came across the following statement: “A study of University of Michigan students found that empathy has decreased by over 40 percent since the 1970s.” How could this be? Could my fellow students be implicated specifically in a society-wide collapse in empathy? There was one immediate red flag: the claim was completely unsourced. There was no link, no reference, and no follow-up. Where could the Yale Herald editorial team have found this?
THE STUDY
After Googling a mish-mash of words and phrases, I was able to track down the study that should have been cited. The study is: “Changes in Dispositional Empathy in American College Students Over Time: A Meta-Analysis,” by Sara H. Konrath, Edward H. O’Brien, and Courtney Hsing. As indicated in the title, this study was a meta-analysis: a study that looks at the results of other studies and aggregates them together to look at a wider trend. In this case, the authors tracked empathy as measured by a scale called the Interpersonal Reactivity Index in 72 other studies which were conducted between 1979 and 2009. Already, the error is clear: this was not a study of U of M students, but of students across America over the course of 30 years. The University of Michigan connection comes from the authors: Konrath worked at the Institute for Social Research, while O’Brien and Hsing were students.
Normally, this would be enough to call it. The authors misinterpreted the study, or perhaps didn’t understand what a meta-analysis was. A couple things kept nagging at me, though. For one, the phrase “72 samples of American college students” appears right in the second sentence of the abstract. It’s impossible to miss. Secondly, the phrasing of the claim was specific in ways not reflected in the original paper. The Konrath article doesn’t have a 40% statistic anywhere: the closest are a 48% and 34% decline in two measures of empathy. “The 70s” also caught my attention: only a few studies came from the very end of the 70s. There had to be something at play here other than misinterpretation.
THE TELEPHONE
I was able to track the 40% statistic pretty easily to a press release from the University itself. In it, Konrath is quoted as such: “College kids today are about 40 percent lower in empathy than their counterparts of 20 or 30 years ago, as measured by standard tests of this personality trait.” Obviously, Konrath isn’t misinterpreting her own study, but is simplifying the conclusions of it to make them easier to understand, as many study authors do. This 40% number is quoted in various non-academic sources. That first source also reports the date range as “the 1970s,” rather than “the late 1970s,” which seems to be the standard phrasing. However, I could not find any other sources claiming that the study was of U of M students, rather than conducted by U of M researchers.
Were I to guess, the author of the Yale Herald did something like this: they saw this statistic somewhere (it’s been cited over 1,360 in other papers alone) and noted three things: the 40% statistic, the date range, and the Michigan affiliation. When it came time to finish the article, they were unable to find it again and wrote about it from whatever they remembered. The article isn’t even mainly about a general decline in empathy, but instead a different concept called “compassion fatigue.” At numerous points, it correctly cites academic studies, and it weaves those in with interviews and personal stories quite well. With everything else, it’s easy for one cursory statistic to get lost and slightly twisted.
THE MORAL
The Yale Herald article is quite excellent, in truth, and I’d heartily recommend it to any readers of this blog. Its quality, in fact, drives the point home even further: anyone could make this mistake. College students read so much that it’s easy to hear something once, file it away, and then draw it out over and over again with no memory of where it came from. Over time, the actual fact can become distorted, until it bears little resemblance to the research data it’s drawn from. For this reason, it’s important not only to cite your sources, but to track down any second hand references. Otherwise, you might end up shouldering Wolverines with the blame for a nationwide decline in empathy.