FrameworkArgumentArtificial intelligenceRiskIncentives
A system optimises the objective you wrote, not the one you meant
Brian Christian · The Alignment Problem
The idea
Christian's history of the alignment problem is a sequence of cases in which a system did exactly what it was told and not at all what was wanted: a game-playing agent that found a loophole in the score, a recommendation engine that maximised engagement by serving outrage, a hiring model that learned the biases in its training data. The common structure is that specifying what we want is the hard part, and that an optimiser will find every gap between the specification and the intention. The idea applies to incentive schemes and performance metrics as directly as it applies to models.
Whatever you can measure and reward will be maximised literally, including the gaps between the measure and the goal.
When it works
AI product and risk discussions, designing any metric or bonus scheme, and evaluating a model's objective.
When it may fail
The book is a history and does not offer a method that solves the problem, because none fully exists. The practical response is monitoring and evaluation, which is the Evals insight.
How to use it
For any objective, model or human, write three ways to score well on it while failing the actual goal. If you can find three in five minutes, so will the optimiser.
Where it came from
Book The Alignment Problem · Brian Christian
What the source says. The clearest non-technical account of why specifying what we want is the hard part, as much a research history as an ethics book.
Connected ideas
Says something similar
Day 2 begins when a company manages to proxies and treats decisions as one-way doorsCheck whether the metric is still the goal or has become the goal, and reserve the heavy process for decisions that cannot be undone.
Takes it further
A model trained on a biased outcome will reproduce it at scaleA model predicts the past decisions in its training data; if those were biased, the model is a bias machine with a straight face.
Puts it to work
Without evaluation you have a demo, not a systemBuild the test set before the feature, measure every change against it, and keep a person in the loop where the metric cannot be trusted.
Concepts
Put it to work on L3VLUP
More like this
- FrameworkProductBook
Shipping is an output. Nobody is paying for outputs
Measure what changed for the customer, not what was shipped; a team given an outcome behaves differently from one given a list.
From Escaping the Build Trap · Melissa Perri
- Case studyCreditBook
A capital structure is a set of rights people will litigate over
In distress, a claim is worth what its documents let it argue for, not where it sits on the slide.
From The Caesars Palace Coup · Max Frumes and Sujeet Indap
- Case studyDeals and M&AFilm
A management buyout puts the seller on both sides of the table
In a management buyout the insider is the buyer, so the board's job is to make the market, not take the offer.
From Barbarians at the Gate · 1993
- FrameworkArtificial intelligenceVideo located
A model never sees words, only tokens, which explains most of its odd behaviour
Reason about a model in tokens: that is what it reads, what fills its window and what you pay for.
From Most Devs Don’t Understand How LLM Tokens Work · Matt Pocock