ArgumentCase studyArtificial intelligenceRisk
A model trained on a biased outcome will reproduce it at scale
Cathy O’Neil · Weapons of Math Destruction
The idea
O'Neil's cases are scoring systems pointed at sentencing, hiring and credit. Each learned from historical outcomes, each reproduced the patterns in those outcomes, and each then applied them to millions of people with the authority of a number and none of the accountability of a person. She was a quant who left finance over exactly this, which is why the argument lands harder than most of the genre. For anyone building or buying a model, the question is what outcome it was trained to predict and whose decisions produced that outcome.
A model predicts the past decisions in its training data; if those were biased, the model is a bias machine with a straight face.
When it works
Any model used on people, credit underwriting, hiring tools, and AI governance questions in a bank.
When it may fail
Human decisions were biased before models existed, and a well-audited model can be fairer than the process it replaces. The argument is for scrutiny, not for abandoning models.
How to use it
Before trusting a score, ask what label it was trained on and who assigned that label. A credit model trained on who was approved learns the old approval policy, not who would have repaid.
Where it came from
Book Weapons of Math Destruction · Cathy O’Neil
What the source says. What happens when opaque scoring systems are pointed at sentencing, hiring and credit, written by a quant who left finance for exactly this reason.
Connected ideas
Worth knowing first
A system optimises the objective you wrote, not the one you meantWhatever you can measure and reward will be maximised literally, including the gaps between the measure and the goal.
Says something similar
What banks actually deploy is narrower and more governed than the consumer conversationInstitutional AI is scoped, governed and data-limited; expect to work inside those limits and to be able to explain them.
Says something similar
The vertical AI revolution belongs to firms with proprietary data, not to the tool makersIn any industry, ask who holds the data the models need; the shift favours them more than the vendors of the models.
Says something similar
Precision against recall: which error you can afford is a business decisionDecide which mistake is more expensive before tuning a model; the threshold is a business choice the statistics only express.
Concepts
Put it to work on L3VLUP
More like this
- Case studyCreditBook
A capital structure is a set of rights people will litigate over
In distress, a claim is worth what its documents let it argue for, not where it sits on the slide.
From The Caesars Palace Coup · Max Frumes and Sujeet Indap
- FrameworkArtificial intelligenceVideo located
A model never sees words, only tokens, which explains most of its odd behaviour
Reason about a model in tokens: that is what it reads, what fills its window and what you pay for.
From Most Devs Don’t Understand How LLM Tokens Work · Matt Pocock
- TechniqueArtificial intelligenceVideo located
A prompt is a brief: task, role, background and output format
Write to a model as you would brief a capable junior: the task, their role, the context they lack, and what the output should look like.
From Prompting 101 · Anthropic
- FrameworkCrises and market historyFilm
A rating is a model output, and the model rests on a correlation assumption
Ask what a rating assumes about defaults happening together before trusting what it says about the pool.
From The Big Short · 2015