By: Mason Williams | West Virginia University | December 10, 2025
The latest integration of AI and mass data collection into domestic security and policing represents an ethical problem for modern civil liberties. Supporters claim that data-driven systems offer objective and efficient means of crime prevention. This paper argues the opposite. Predictive policing and surveillance, used by companies like Palantir and tools such as COMPAS, will ultimately cause more harm than good. This will create a failure in public trust, deepen systemic bias, and undermine fundamental rights, especially in the United States.
The main issue of this technology is that it enforces the normalization of mass domestic surveillance and reinforces historical injustices and inequality. This is because the data being used is already from flawed and biased datasets. This critique is supported by the analysis of two philosophical frameworks. The first, Clinton Castro’s Risk of Misclassification, and the second, Renée Jorgensen’s interpretation of the right to be treated as an individual. By applying these concepts, there are two solutions rather than fixing algorithms. The first one is to add human oversight so it’s not just an AI that is judging a human. The second solution is to completely reject the use of AI in surveillance and the judicial system.
The primary ethical issue with predictive policing is its failure to achieve fairness. Instead, it replicates and amplifies existing social bias. The Correctional Offender Management Profiling for Alternative Sanctions system, or COMPAS for short, is designed to assist judicial decisions such as bail, parole, and sentencing. It relies heavily on historical data that is already flawed since it contains decades of unequal policing and racial discrimination. This, in turn, guarantees biased outcomes. One of Castro’s arguments in “What’s Wrong with Machine Bias” is that these technologies and algorithms are biased against marginalized groups, whom they identify as high-risk repeat offenders at nearly twice the rate of white defendants (Castro 408). This is especially true against black defendants, as shown in the case of Borden and Jones.
The machine bias case is best demonstrated in the comparison of Brisha Borden, Sade Jones, and Varnon Prater. This was originally reported by ProPublica and dissected by Castro. Borden and Jones, who are both black, were arrested for petty theft. The value of the items was determined to be less than one hundred dollars; however, COMPAS gave them a high and medium-risk score. Vernon Prater was the suspect in multiple armed robberies, was arrested for stealing items valued at eighty-six dollars, and received a low-risk score (Castro 405). Prater went on to re-offend, while Borden and Jones didn’t and completed their probation. This case shows the inequality of the COMPAS system. It uses statistical generalization that punishes certain people, such as neighborhood, parental arrest history, and socioeconomic status, instead of the individual themselves.
Castro uses the risk of misclassification as an ethical tool to analyze these wrongs. The issue is not that the algorithm is wrong but that the person was exposed to a risk of being misclassified that they could reasonably reject (Castro 416). Castro’s theory also considers the broader societal context. Black people already face disproportionate costs because of systemic bias in housing, employment, and law enforcement. So, he considers their threshold for risk to be lower. The COMPAS score is therefore biased since it imposes a greater burden on Burden and Jones than the risk imposes on Prater. The system disproportionately distributes the cost of its errors onto those who are already burdened by societal prejudices.
This issue of algorithmic bias, as shown by Clinton Castro, goes beyond stating that the algorithms are inaccurate. It focuses on the unfair and disproportionate risk of misclassification imposed on already marginalized groups. This is rooted in the broader societal context of cumulative disadvantage. Black defendants such as Brisha Borden and Sade Jones are not starting in the same position when their risk is assessed. They are facing the pre-existing disproportionate costs in life, such as housing, employment, and interaction with law enforcement, due to systemic bias. This means that any system that adds an extra layer of disadvantage, such as the risk score from COMPAS, pushes them further into vulnerability than it would for a white defendant, and like Vernon Prater, who is benefiting from it.
The risk for marginalized groups is higher because of the difference in their societal starting points. This is why the COMPAS score is a clear example of systemic bias, where the system disproportionately imposes the cost of its errors onto those who are already prejudiced. The risk of being misclassified as a repeat offender is not just a statistical error but an ethical error. It favors the status quo of inequality because it guarantees biased outcomes from the data it is already trained on, which contains decades of racial discrimination and unequal policing.
Looking beyond the risk of misclassification now, predictive policing and mass data collection violate the right to be treated as an individual. Renée Jorgensen expresses this right as she says it entails “A claim to not be subject to extra burdens simply on account of one’s social identity” (67). This is similar to Castro’s take as well. This right is fundamentally flawed with the mechanism of predictive policing because the algorithms have to treat certain individuals as members of a risk-prone group. This goes against the idea of human autonomy because it treats people as if they’re not accountable for their own actions. Although COMPAS does not explicitly use race, it relies on factors such as a parent’s arrest history, residence, and socioeconomic background, which have close ties with race and status, which are outside of a person’s control. This makes it so crime belongs to a reference class that imposes extra burdens based on a demographic, which violates the right to be treated as an individual (Jorgensen 63).
The entire idea of predictive policing violates this right because the algorithms must treat certain individuals as members of a risk-prone group based on statistical generalization by necessity. This is contrasted to autonomy since it assesses a person’s future based on the actions of their demographic group rather than their own actions. COMPAS may not explicitly input race, but it does rely on factors that are closely tied to race and socioeconomic status, such as parents’ arrest history, residence, and socioeconomic background. These factors are out of an individual’s control, but they are used to categorize and impose burdens on them. When algorithms use these variables, the concept of crime is forced upon a reference class, which punishes the demographic rather than the individual. The algorithm is hidden from the public, which violates the principle that the law must announce clear expectations that citizens can avoid violating, according to Jorgensen. This failure to uphold equal protection for the sake of efficiency is what sacrifices public trust and legitimacy between citizens, the state, and, of course, the police.
The mass data collection of companies such as Palantir pushes this issue to the scale of domestic surveillance, which risks the normalization of a security state that undermines the civil liberties that Americans and other “free” countries hold so close. Palantir has gained contracts for data analytics and strategic subject initiatives, which are comprised of crime reports, service requests, and conviction data. This creates a pervasive network of surveillance. We know the data they’re using is flawed by enforcement bias, which means that, just like COMPAS, it will inevitably project past injustices into the future (Jorgensen 73). Jorgensen says that the law must announce expectations that citizens can avoid violating. However, the basis for suspicion is hidden within an unclear algorithm. This makes it so the law becomes retroactive and fails to respect individual autonomy (Jorgensen 74). This is what will break the trust between citizens and the state since it sacrifices the promise of equal protection for efficiency.
The main defense for predictive policing and the expansion of surveillance networks, such as Palantir, is that it will improve efficiency and objectivity. However, these claims do not hold up under ethics. By automating risk assessment, the justice system is supposed to become more efficient, which frees resources and expedites decisions such as bail and sentencing, in the case of COMPAS. This efficiency, however, is gained at the expense of Americans’ rights. Jorgensen’s belief is that sacrificing the promise of equal protection for efficiency is what breaks trust between citizens and the state. Castro also points out that the overall value of algorithmic prediction in areas such as street crime is already low because of the high rates of false positives and the ease of prediction using simpler, less intrusive methods. This means that the efficiency gained is negligible when pitted against the societal cost of normalizing mass surveillance and reinforcing systemic bias.
The ethical frameworks of the risk of misclassification and the right to be treated as an individual reject the idea that the algorithms only require improvement. The issue is not that the tool needs perfecting but that it treats humans as data or ones and zeroes. It has no legitimacy in a justice system that upholds individual rights since it doesn’t see every aspect that makes up a person’s life. This could be anything from financial struggle to their childhood. So, this leaves two options: introducing human oversight or the complete rejection of predictive policing and AI tools to determine one’s freedom.
Introducing human interaction or human oversight into the process reintroduces the concept of individuality. Since an algorithm sees someone as a set of variables, it strips them of their uniqueness and agency. Human review can override algorithmic scores based on individual circumstances such as economic status or race. It can improve the lack of humanity that is used when a computer decides on a person’s freedom. Human intervention would serve to moderate the inherent bias of data sets that algorithms use to amplify social bias. This would ensure that decisions about a person’s life and freedom are not made by this algorithm that is trained on historical inequality.
For human review to reintroduce this individuality and agency, the overseer, possibly a judge, jury, or a parole officer, must be given full access to the score and its contributing factors that the algorithm gives. They must be given access to all factors outside of their control, such as a parent’s arrest history, residence, and socioeconomic background. This transparency would give the overseer the ability to recognize and override a score that is unquestionably based on systemic bias rather than their individuality. This would also give the overseer accountability for the defendant instead of an algorithm that can’t be punished.
The second option is the most invasive and possibly would have the biggest outcome, which would be the complete rejection of predictive policing. Since algorithms are making high stake decisions that could be devastating to one’s life, this option would be the safest. Castro says that the value of a prediction in street crime is low due to high rates of false positives and the relative ease of prediction using simpler methods. Jorgensen says that such applications are ruled out because they always track socioeconomic disadvantages and impose disproportionate burdens (74). If a predictive algorithm cannot satisfy the conditions of control and transparency, it cannot be ethically deployed. Since the properties correlated with street crime are measures of socioeconomic status that people cannot reasonably control, using them to apply criminal punishment is unfair. So, this would completely get rid of the algorithm due to its inherent unfairness, no matter what.
Attempting to fix or improve these biased algorithms is pointless since it would legitimize a flawed premise due to the data itself reflecting long histories of unequal bias. The use of AI and mass data collection in policing, such as the goal of Palantir’s systems, fundamentally opposes the right to autonomy and equal treatment under the judicial system. An algorithm cannot determine a person’s freedom and life since it is not human itself. This would be inherently inhumane as they cannot comprehend individuality and the hardships of systemic inequality. Introducing human oversight would keep data collection; however, it can still keep a defendant’s individuality and agency into perspective in their sentencing. It would also give accountability to the person reviewing the data that an algorithm would give. Completely rejecting the idea of predictive policing would completely take away the ability to use systemically biased data into account for a person’s freedom. Therefore, instead of continuing the use of the systemically flawed tools of surveillance and predictive policing, the justice system must choose between human judgment and complete human involvement to protect an individual’s liberty, freedom, and equality.
Works Cited
Angwin, Julia, et al. “Machine Bias.” ProPublica, 23 May 2016,
www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Castro, Clinton. “What’s wrong with machine bias.” Ergo, an Open Access Journal of Philosophy, vol. 6, no. 20201214, 11 July 2019,
https://doi.org/10.3998/ergo.12405314.0006.015
Jorgensen, Renée. “Algorithms and the individual in criminal law.” Canadian Journal of Philosophy, vol. 52, no. 1, 11 Oct. 2021, pp. 61–77, https://doi.org/10.1017/can.2021.28