FrameworkTechniqueArtificial intelligenceSoftware engineering
The context window is not the constraint people think it is; attention across it is
NetworkChuck · Why LLMs Get Dumb (Context Windows Explained)
The idea
Everything a model knows at answer time is in its context window, so the obvious limit is the window's size. NetworkChuck's explanation is about the less obvious one: as the window fills, the model's attention across it degrades, and material in the middle of a long context is used less reliably than material at the start or end. The practical consequences are that a bigger window is not a licence to dump everything in, that what you put in front of the model should be curated and ordered, and that the engineering discipline of deciding what goes in the window matters more than the window's headline size.
A long window degrades before it fills; curate and order what goes in rather than relying on size.
When it works
Designing any system that feeds documents to a model, and explaining why a model missed something that was in its context.
When it may fail
Attention behaviour varies by model and improves between generations, so the point of degradation moves. Measure it for the model you use rather than assuming.
How to use it
For a retrieval system, measure answer quality as the number of passages in the context rises. The point where it starts to fall is your real window, and it is smaller than the advertised one.
Where it came from
Video Why LLMs Get Dumb (Context Windows Explained) · NetworkChuck
What the source says. NetworkChuck explains why attention degrades across a long window, and how to work within it.
Connected ideas
Worth knowing first
A model never sees words, only tokens, which explains most of its odd behaviourReason about a model in tokens: that is what it reads, what fills its window and what you pay for.
Puts it to work
Without evaluation you have a demo, not a systemBuild the test set before the feature, measure every change against it, and keep a person in the loop where the metric cannot be trusted.
Says something similar
A prompt is a brief: task, role, background and output formatWrite to a model as you would brief a capable junior: the task, their role, the context they lack, and what the output should look like.
Concepts
Put it to work on L3VLUP
More like this
- Company insightArtificial intelligenceLetter located
Inference has overtaken training, and the flywheel runs on tokens
A compute vendor's case is a flywheel from agents to tokens to compute; the buyer's case is the return on that compute, and both have to be read together.
From NVIDIA's fiscal 2026 letter from Jensen Huang · Jensen Huang
- ArgumentArtificial intelligenceBook
A model trained on a biased outcome will reproduce it at scale
A model predicts the past decisions in its training data; if those were biased, the model is a bias machine with a straight face.
From Weapons of Math Destruction · Cathy O’Neil
- Career lessonCareersDocumentary
A skill is only worth what the market structure around it allows
Separate what you can do from the structure that makes it valuable, and watch the structure.
From Floored · 2009
- FrameworkArtificial intelligenceBook
A system optimises the objective you wrote, not the one you meant
Whatever you can measure and reward will be maximised literally, including the gaps between the measure and the goal.
From The Alignment Problem · Brian Christian