FrameworkArtificial intelligence
A model never sees words, only tokens, which explains most of its odd behaviour
Matt Pocock · Most Devs Don’t Understand How LLM Tokens Work
The idea
A language model reads tokens, pieces of text that are often whole common words and often fragments of rare ones. Matt Pocock takes tokenisation from characters to subwords and shows where unusual words go wrong. Three consequences follow for anyone building on or using a model. It miscounts letters because it never saw the letters. Context has a size because the window is counted in tokens. And usage is billed in tokens, so the cost of a prompt depends on its tokenisation, not its word count. Understanding this removes most of the mystery from a model's failures and most of the surprise from its bills.
Reason about a model in tokens: that is what it reads, what fills its window and what you pay for.
When it works
Explaining how a language model works, estimating the cost of an AI feature, and debugging a model that fails at a character-level task.
When it may fail
Tokenisers differ between models, so an estimate for one is wrong for another, and the relationship between tokens and meaning is not simple. The frame explains behaviour; it does not predict quality.
How to use it
Before estimating the cost of an AI workflow, tokenise a representative input and output and multiply by the price per token. Then compare with the word count to see how far the two diverge for your data.
Where it came from
Video Most Devs Don’t Understand How LLM Tokens Work · Matt Pocock
What the source says. Matt Pocock takes tokenisation from characters to subwords, and shows where unusual words go wrong.
Connected ideas
Takes it further
The context window is not the constraint people think it is; attention across it isA long window degrades before it fills; curate and order what goes in rather than relying on size.
Says something similar
Inference has overtaken training, and the flywheel runs on tokensA compute vendor's case is a flywheel from agents to tokens to compute; the buyer's case is the return on that compute, and both have to be read together.
Concepts
Put it to work on L3VLUP
More like this
- ArgumentArtificial intelligenceBook
A model trained on a biased outcome will reproduce it at scale
A model predicts the past decisions in its training data; if those were biased, the model is a bias machine with a straight face.
From Weapons of Math Destruction · Cathy O’Neil
- TechniqueArtificial intelligenceVideo located
A prompt is a brief: task, role, background and output format
Write to a model as you would brief a capable junior: the task, their role, the context they lack, and what the output should look like.
From Prompting 101 · Anthropic
- Career lessonCareersDocumentary
A skill is only worth what the market structure around it allows
Separate what you can do from the structure that makes it valuable, and watch the structure.
From Floored · 2009
- FrameworkArtificial intelligenceBook
A system optimises the objective you wrote, not the one you meant
Whatever you can measure and reward will be maximised literally, including the gaps between the measure and the goal.
From The Alignment Problem · Brian Christian