FrameworkArtificial intelligenceVideo located
A model never sees words, only tokens, which explains most of its odd behaviour
Reason about a model in tokens: that is what it reads, what fills its window and what you pay for.
Explaining how a language model works, estimating the cost of an AI feature, and debugging a model that fails at a character-level task.
From Most Devs Don’t Understand How LLM Tokens Work · Matt Pocock