In 2018, Amazon discovered that its AI-based recruiting system systematically penalised women's CVs. The algorithm had been trained on ten years of hiring data — in a male-dominated company. It had "learned" that being a woman was a negative factor. Amazon abandoned the system, but the case revealed a structural problem: algorithmic biases are not errors, they are mirrors of the inequalities in the society that produces them.
COMPAS, an algorithm used in American courts to assess prisoners' recidivism risk, is perhaps the most well-known case. A ProPublica investigation in 2016 showed that the system systematically overestimated risk for African-American prisoners and underestimated it for white ones. Decisions about people's freedom were being made by an algorithm that no one could inspect, built by a private company, with opaque criteria.
Algorithmic biases have three main sources. First is training data: if the data reflects historical inequalities (and it almost always does), the algorithm will perpetuate them. Second is the choice of variables: deciding what to measure and what to ignore is a political choice disguised as technical. Third is the objective function: optimising for efficiency produces different results from optimising for equity.
The problem is compounded by opacity. Many decision-making algorithms are "black boxes": they produce outputs without explaining how they arrive at them. When an algorithm denies a mortgage, rejects a CV or determines a prisoner's sentence, the person involved has the right to know why. The European GDPR provides for a "right to explanation", but its enforcement remains vague.
What can we do? Independent algorithmic audits, transparency in training datasets, diversity in development teams, and above all: consciously deciding when an algorithm is appropriate and when it is not. Not everything needs to be automated. Some decisions — those concerning freedom, health, education — require human judgement, context and compassion that no algorithm can replicate.