Written by Radi Akbar
After months of protest from the DC fanbase, Warner Brothers finally released the Snyder cut of the Justice League. However, one might be worried about viewing this movie. First of all, this new director’s cut is 4 hours long and secondly many audiences were worried if the new movie is going to be as bad, or worse, as the original. One way to find the value of this movie is from the ratings given from sites like Rotten Tomatoes or IMDB. Currently, the Rotten Tomato score for the Snyder cut is 71%, implying that 71% of viewers enjoyed this film, but this is misleading. Keep in mind that only 279 people have reviewed the film, so one might assume that the sample size is too small to judge the film. So, how can someone properly evaluate these ratings?
One of the methods to evaluate ratings is through using the beta distribution. What the beta distribution essentially does is give all possible ratings given the number of positive and negative reviews (Cook, 2017). As the number of reviewers increases, the distribution is more centered at it’s true rating. This is because the increased number of reviewers gives new information about the product. So what does the 279 reviews imply? The 71% rating becomes more clear because the distribution is more concentrated at the true value. Thus, there is a high chance that 71% of the viewers enjoyed the movie.
Figure 1. Showing beta(1,0), beta(36 ,14), and beta(71, 29).
Note that K is the number of positive reviews. (Bengfort, 2017)
However, this statistical method of evaluation is not without its limitations. One of the problems of product reviews is the self-selection bias. This bias occurs when reviewers voluntarily give their opinion regarding the product that they purchased. The problem with self-volunteer reviews is that the sample is no longer random. Usually, samples are obtained by randomly asking every customer about their experience on the product. If the sample is composed of self-volunteer reviews, it does not take into account customers who did not bother to give a review in the first place.
A study showed there are two major self-selection biases within product reviews. First, there is acquisition bias, this is when reviews mostly come from customers who have a tendency of reviewing a product. Second, there is underreporting bias, when customers with extreme opinions are more likely to write a review compared to their moderate counterparts. The study concluded that customers are aware of these self-selection biases, but are unable to correct them due to bounded rationality (Hu, Nan, et al, 2017). In economics, people are rational, meaning that given a set of choices, they would choose the one that is optimal. Bounded rationality implies a constraint whether it is in information or time. In this case, customers do not have enough information about the reviews to infer the true quality of the product.
Furthermore, products like movies or books are subject to the survivorship bias. This bias occurs when only well-known products are reviewed, so popular movies like Avengers: Endgame can have millions of reviews, while a less known indie film might receive less reviews.
Mathematician Hannah Fry explained that the stigma ‘Hollywood ruins good books’ is false because people only compare good books and movies only. She explained that when people ignore the larger dataset and only focus on specific subsections, they can extract some misleading inferences. In this case, people ignore bad books that have bad film adaptations or mediocre books with mediocre film adaptations. Why? Because they don’t attract the same attention as a good movie or book would. So, in the larger dataset, there is no relationship between good movie adaptations and books (Haran, 2018). The same idea applies to product reviews, well-known products get more reviews and more concentrated beta distributions, while less-known products get fewer reviews and more spread out beta distributions.
So, what can customers do in order to find out if the Snyder cut of the Justice League is worth watching? One way is to watch comprehensive reviews on the internet. These reviews can give information that is hard to find within movie ratings and more importantly overall impression of the product. This also applies to other products like books, tech, etc. However, these comprehensive reviews are not without the same problem. First of all, these reviews are still voluntary, meaning the reviewer can have extreme opinions about the product or be an avid fan of the creator. Second, comprehensive reviews are still subject to survivorship bias because these reviews are intended to earn viewerships rather than properly evaluating products.
So, if beta distributions and comprehensive reviews are subject to biases, then what’s a bias-free-way to evaluate the true value of a product? Statistician George E.P. Box once said that “all models are wrong, but some are useful”. In the real world, there are too many factors or hidden information that prevents people from finding the true value of a product, but these models like the beta distribution and comprehensive review give an intuition on evaluating the value of a product. The beta distribution does not necessarily filter bias, but it does give a general idea on how many reviews are needed for the rating to concentrate on a it’s value. Comprehensive reviews can also give important information about the product, especially when the reviewer is an expert and knows the needs of a general consumer. Models are not meant to simulate reality, but act as a map to navigate a very complicated world. As such, the most important factor in finding the value of a product is human intuition.
References:
Cook, John D. “A Bayesian View of Amazon Resellers: Beta-Binomial Model.” John D. Cook | Applied Mathematics Consulting, 14 July 2017, www.johndcook.com/blog/2011/09/27/bayesian-amazon/.
Bengfort, Benjamin. “Computing a Bayesian Estimate of Star Rating Means.” District Data Labs, Medium, 17 Dec. 2017, medium.com/district-data-labs/computing-a-bayesian-estimate-of-star-rating-means-651496a890ab.
Stephens, Matthew. “Bayesian Inference for a Binomial Proportion.” FiveMinuteStats, 25 Jan. 2017, stephens999.github.io/fiveMinuteStats/bayes_beta_binomial.html#overview.
Haran, Brady. “Does Hollywood ruin books? – Numberphile”. YouTube, edited and animated by Pete McPartlan, 28 Aug. 2018, https://www.youtube.com/watch?v=FUD8h9JpEVQ&ab_channel=Numberphile.
Sanderson, Grant. “Binomial distributions | Probabilities of probabilities, part 1”. YouTube, music by Vincent Rubinetti, 15 Mar. 2020, https://www.youtube.com/watch?v=8idr1WZ1A7Q&ab_channel=3Blue1Brown.
Sanderson, Grant. “Why “probability of 0” does not mean impossible | Probabilities of probabilities, part 2”. YouTube, music by Vincent Rubinetti, 12 Apr. 2020, https://www.youtube.com/watch?v=ZA4JkHKZM50&t=0s&ab_channel=3Blue1Brown.
Hu, Nan, et al. “On Self-Selection Biases in Online Product Reviews.” MIS Quarterly, vol. 41, no. 2, 2017, pp. 449–471., doi:10.25300/misq/2017/41.2.06.