Written by Connor Zahler
INTRODUCTION
It’s not a stretch to assume that most University of Michigan students have heard of, or even consulted, U.S. News and World Report. Since 1983, they’ve put out a ranking of colleges and universities across the United States. Michigan, of course, often ranks highly. They don’t stop at undergraduate programs, though, with similar rankings for graduate programs like med school and law school.
Despite their prestige, the program is not without controversy and allegations of gamifying university operations. Readers may remember Columbia University tumbling from second to eighteenth following revelations that they falsified data for a higher ranking. More recently, multiple law schools have announced that they will no longer submit data for ranking. One may assume this is a case of being mad about a poor ranking until they see the names attached, which include Yale, Harvard, and the University of Michigan. The reasoning behind it is, surprisingly, directly related to quantitative methods.
BACKGROUND: WHO CARES ABOUT THESE RANKINGS?
U.S. News and World Report have formed the gold standard for university rankings for decades. Originally, they were the only place that compiled and analyzed a wide variety of data on law schools. Students turned to them to help them choose where to apply and eventually attend. Colleges often make hay out of their rankings, littering the numbers across advertising and promotional materials. At a certain point, the differences between universities become less about qualitative differences and more about prestige. In short, there’s a lot riding on these rankings. Why, then, are so many law schools pulling out?
AN EXERCISE IN CLOSE READING
To get to the quantitative bottom of this, let’s take a look at the message from U-M’s own Dean of the Law School, Mark West.
“As a public institution, serving the public interest has always been central to our mission. Over time, I increasingly have come to believe that the U.S. News law school rankings no longer serve the public interest.”
This statement starts pretty simple. UM Law is an organization that serves the public, and participating in these rankings does not serve the public, so they’ve stopped participating. West goes on to explain that discussions had been ongoing for years but became decisive once Yale (led by a U-M Law grad) took action.
“U.S. News law school rankings provided valuable information for consumers—most importantly, students—that had not previously been widely available. This is no longer true…”
For over a decade, law schools have been required to publish information about their student body, spending, governance, and eventual job placements by the American Bar Association. West goes on to criticize US News for not keeping up with changing standards before getting to the real meat of what we’re interested in:
“U.S. News rankings are opaque in both their methodology and content (much of the data is not publicly revealed), and unavailable to those uninterested in paying to scrutinize them.”
Aha! At the crux of the matter are questions about methodology and the transparency (or lack thereof) of data. Let’s see what else West has to say.
LURKING ALGORITHMS
The most heavily weighted component of the rankings comes from academic opinion polls: U.S. News surveys select administrators and faculty members at every law school to collect their opinions about their perceptions of the reputation of every other law school. The collected information might be interesting, but it is not based on a rigorous survey instrument, and even if it were, it should not guide decision making for prospective students (or anyone else)…[w]hile Michigan has consistently resisted the pressure to take actions that are contrary to our mission, the demands of the U.S. News algorithm always lurk in the background.
If you changed some words around, this could be a critique of any number of big tech companies you read about in a QMSS class. US News puts disproportionate weight on a poll that is not collected systematically or randomly. In other words, it’s not the best poll, and it’s being used to make important choices. Using bad data sources has a wealth of consequences (for example, severely overestimating the number of people executed for witchcraft), as any QMSS 201 alum knows.
US News is also making the classic mistake of changing a measure to a goal. Rather than doing their best to serve their constituents and hoping that will correspond to a good ranking, schools are now trying to predict the whims of the algorithm used to predict rankings. West argues that many of these changes involve pushing costs onto students. No matter the veracity of this claim, it points to the same concern we see in similar circumstances: worries that inhuman algorithms are poorly guiding human behavior.
“Changes to the formula often are announced after the fact, or simply never explained at all…[m]oreover, U.S. News neither vets nor authenticates the data. This situation presents, at best, inequitable presentation of data and at worst, an unregulated opportunity for manipulation. In addition, if a law school wants to see the full set of aggregate data that others submit, it must pay U.S. News for access.”
Anyone else getting deja vu? This is, again, the same story we always hear about complex algorithms that are kept private. Their methodologies are opaque, which means that there are lots of opportunities for error and outright manipulation. Essentially, US News is collecting tons of data without much investigation (data that cannot be seen without paying), throwing it into a machine that comes with little explanation, and churning out numbers that have a massive impact on students and schools. This machine is in large part based off of a poll that wouldn’t pass muster in an undergrad class. If these claims are true, no wonder that so many schools are pulling out.
OUT OF LINE?
When you’re sitting in a lecture, it can be hard to see the applications outside of that class, let alone to a billion-dollar enterprise that determines the course of the judicial system. It may also be hard to believe that a QMSS 201 case study is revealing errors in a heavily relied-upon institution of higher education. Truth is stranger than fiction, though, as this example shows us. There’s a hidden quantitative element to a lot of news stories you see, and UM Law pulling out of US News and World Report’s ranking is no different.