The Context Window: What the Model Can Actually See

The context window is everything the model can see at once: your instructions, the conversation so far, the documents you pasted, and the answer being written. It is not memory. Nothing survives the end of the request.

Fifth in a series on how language models actually work.

One window, shared by everything

A model advertising a 200,000-token window is describing a single shared budget. The system prompt, chat history, retrieved documents, tool definitions, tool results and the response all come out of the same allowance. Nothing is free, and the response competes with the input.

This catches people out when tools are involved. Every tool the model can call is described inside the window, and those descriptions are always present whether or not the tool is used. Twenty verbose tool definitions can consume a serious fraction of the budget before the user has typed anything.

Long does not mean uniformly good

A large window means the model will accept that much text. It does not mean it uses all of it equally well. Measured across many models, retrieval of a specific fact is strongest when the fact sits near the beginning or the end of the context and weakest in the middle — the “lost in the middle” effect.

The engineering response is simple and effective. Put instructions at the top and the actual question at the bottom, so the thing you most want obeyed is in a strong position. And prefer selecting five relevant paragraphs over pasting fifty and hoping. A smaller, better-chosen context routinely outperforms a larger one, and costs less.

There is no memory between requests

Each API call is self-contained. A chat interface feels continuous because it silently resends the earlier turns every time. Nothing you said is remembered as conversation state. A provider may cache the computation over a repeated prefix, as described below, but that is a performance optimisation rather than memory, and it changes nothing about what the model can recall.

Two things follow. Conversations get progressively more expensive, because each turn resends everything before it. And when a long chat starts forgetting, it is usually because older turns were trimmed to fit — the information did not decay, it was removed.

Anything that must genuinely persist has to live somewhere you control — a database, a file, a summary you re-inject deliberately. That is the whole basis of agent memory design, and it is an ordinary storage problem wearing unusual clothes.

Caching changes the economics

Because the expensive part of a long prompt is processing the same prefix over and over, providers let you cache it. Keep the stable material — system prompt, tool definitions, a reference document — at the front and unchanged, and repeat calls reuse that work at a large discount.

This rewards a specific layout: stable content first, variable content last. Putting a timestamp at the top of an otherwise fixed prompt invalidates the cache on every call. Moving it to the bottom costs nothing and preserves the discount.

What to take away

The context window is a working desk, not a filing cabinet. It is shared by every part of the request, it is not recalled evenly, it empties completely between calls, and its front section is the cheapest place to put anything that does not change. Design around those four facts and long-context systems become predictable rather than mysterious.

Earlier in this series: how a language model works, what tokens are, how attention decides what matters, and what temperature really controls.