Algorithmic bias is not an abstract concept: it has concrete, measurable consequences on people's lives. In this lesson we examine three domains where the impact is particularly severe: the justice system, the labour market and the healthcare system.
Justice: the COMPAS case. In the United States, many courts use "risk assessment" software to help judges decide on bail, parole and sentence severity. COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is the most well-known. In 2016, a ProPublica investigation revealed that COMPAS assigned Black people nearly twice the recidivism risk of white people, even when their criminal histories were comparable. The company that produced COMPAS, Northpointe, contested the analysis, arguing that the system was "calibrated" correctly. Both were right — because, as mathematicians later demonstrated, different definitions of "fairness" can be simultaneously incompatible. The question "is it fair?" has no technical answer: it is a political choice about which type of fairness to prioritise.
Hiring: the Amazon case. In 2018, Reuters revealed that Amazon had developed an AI system for CV screening that systematically penalised women. The reason was simple: the system had been trained on CVs from employees over the past 10 years, a period during which Amazon — like most of the tech sector — was predominantly male. The model had "learned" that CVs with words like "women's" (e.g. "captain of women's chess club") were negatively correlated with hiring. Amazon scrapped the project, but the case illustrates a general principle: a system trained on the past will tend to replicate the past.
Healthcare: the Optum algorithm. In 2019, a study published in Science revealed that an algorithm used by American hospitals to allocate healthcare resources systematically discriminated against Black patients. The algorithm used healthcare costs as a proxy for the severity of medical conditions. But, due to structural inequalities in access to care, Black people tended to spend less on healthcare — not because they were healthier, but because they had less access. The result: given equal health conditions, the algorithm allocated fewer resources to Black patients. A measurement bias with potentially lethal consequences.
These cases are not exceptions: they are the rule. Every time an algorithmic system is used to make decisions about people — and especially when these decisions involve historically marginalised groups — the risk of discrimination is real and systemic. The solution is not to stop using technology, but to govern it with awareness, transparency and accountability.