Key takeaways
Forty messages into a long chat, the model starts ignoring the instruction you gave it in the first one. It hasn’t forgotten in any human sense. The instruction has fallen out of its context window.
That’s the amount of text a model can read at one time, counted in tokens. Everything in one request has to fit inside it: the system prompt, the chat so far, any documents you pasted and the answer the model is about to write. Think of it as a budget, not a memory, and most of a chatbot’s odd behaviour makes sense.
This follows my explainer on LLMs, which covered tokens. Here I look at the space they live in.
One budget, four claims on it
Anthropic’s docs define the window as all the text a model can reference when generating a response, including the response itself. People miss that last part. The answer comes out of the same pool as the input. So 195,000 tokens of documents in a 200,000-token window leave room for an answer of 5,000 tokens and no more.
Four things compete for the space:
The window is the model’s working memory for one request. Its training is what it knows in general. The window is what it can see right now, and only the window resets each time you press enter.
Why does a long chat lose its beginning?
Most chat apps manage the window first in, first out. As the running total passes the limit, the earliest messages drop out to make room for new ones. The model can’t refer to text it can no longer see.
So this is a layout choice, not a mystery. Anything that must hold for the whole session belongs in the system prompt, which usually stays pinned. Don’t put it in an early message that will scroll away.
The model hasn’t forgotten the start of the chat. It just can’t see it any more. You can fix that from your side.
Bigger costs more
Once you see the budget, you want a bigger one. But three costs grow with what you send.
Money. API use is billed per token, and the rate isn’t flat. Past 200,000 tokens, a request to Claude Sonnet 4 goes from $3 to $6 per million input tokens and from $15 to $22.50 for output.
Speed. Before writing a word, the model must read everything you sent. The attention mechanism from Attention Is All You Need (2017) compares every token with every other. So the work grows faster than the input, and the memory it needs grows with the length too.
Recall. This one’s the least obvious. The 2023 study Lost in the Middle found that models use information best at the very start or end of the input, and much worse in the middle. That held even for models built for long context. The RULER benchmark found only about half the models claiming 32,000 tokens held up at that length. Anthropic calls the decline context rot.
Five years ago a window of 2,000 tokens was normal. Several Claude models, including Opus 4.8 and Sonnet 4.6, now hold 1 million tokens, and others 200,000. Google says Gemini takes 1 million or more.
A million tokens is about 750,000 words, longer than most novels. But the number on the box is a limit, not a target. Filling it is exactly what brings on those three costs.
How do you fit what matters?
Put the instructions and facts the answer depends on at the top or the bottom of the prompt. Never leave them in the middle.
Keep the parts that don’t change, like a fixed system prompt or reference document, at the start. Then prompt caching can reuse them. Anthropic bills cache reads at a tenth of the normal input price, and they speed up the first token on long documents.
And send the smallest set of useful text, not everything you have. Anthropic’s own guidance for builders treats context as a finite resource with diminishing returns.
Don’t ask how much fits in the window. Ask how little you can send and still get the answer right.
When should you use retrieval?
At some point no window is big enough. A hundred thousand support tickets, or a wiki that changes daily, can’t be pasted. Even if they fit, you’d pay to read all of it. Then you’d ask the model to find a fact in the middle, where it’s weakest.
The other way is to fetch only the passages that bear on the question. That was introduced as retrieval-augmented generation in 2020. It’s now the standard way to answer from private or changing data.
A big window suits one long contract, or one codebase that must be seen whole. Retrieval suits finding the right 5,000 tokens among millions. I take it apart in what RAG really is.
Frequently asked questions
What is a context window in simple terms?
It's the amount of text a language model can read at one time, counted in tokens (a token is roughly three-quarters of a word). Everything the model needs for one reply has to fit inside it: the instructions, the chat so far, any files you paste, and the reply itself. Think of it as the model's working memory for one request, not a long-term store.
How big are context windows in 2026?
Frontier models run from about 200,000 tokens to 1 million. Several Claude models and Google's Gemini both reach 1 million tokens. That's enough for a long book or a mid-sized codebase. A few research systems advertise more. But the usable amount is normally smaller than the advertised number, because recall drops well before the hard limit.
Why does the model forget what I said at the start of a long chat?
It doesn't forget so much as lose sight of it. Most chat apps run the window first in, first out. Once the running total of tokens passes the limit, the earliest messages drop out and the model no longer sees them. It can't use text it can't see.
Does a bigger context window always give better answers?
No. A larger window lets you include more, but every extra token costs money and adds delay. Accuracy also tends to fall as the window fills, which vendors now call context rot. Picking the right few thousand tokens usually beats dumping in hundreds of thousands and hoping the model finds the part that matters.
What is the difference between a context window and model memory?
The context window is per request. It resets every time and only holds what you send. Product memory features fake continuity by saving notes about you and adding them back into the prompt on the next request. So those notes use up the same window. The model still starts each request with only what fits in the budget.
Three-minute lesson on this What AI actually is Marketing course, basic. Free, no sign-up.