What Is a Context Window (n_ctx)? How Much an AI Model Can Hold at Once

Run a model locally and you will meet n_ctx almost immediately, usually in a settings panel asking how big a context window you want. That single number decides how much text the model can read and write in one go, and it quietly sets how much memory the whole session eats.

Quick Answer

A context window (n_ctx) is the maximum number of tokens a model can handle in a single pass, counting both your prompt and the reply it generates. Bigger windows hold more, but they cost memory: pushing n_ctx to 8,192 on a 7B model adds roughly 1GB to 2GB of VRAM use on top of the model itself.

What the context window actually holds

A token is a chunk of text, very roughly three-quarters of an English word. The context window is the model's working memory for one conversation turn: everything it can "see" at once. Crucially, the input and the output share that budget. If your window is 4,096 tokens and your prompt fills 3,000 of them, the model has only about 1,096 tokens left to answer before it runs out of room and starts forgetting the start of the conversation.

That is why long chats with a small window begin to lose the thread. The earliest messages fall out of the window to make space for new ones. Raising n_ctx is what lets a local model hold a long document, a big code file or an extended back-and-forth without dropping the beginning.

Why a bigger window costs memory

The window is not free. The model keeps a running cache of every token in the window so it does not have to recompute earlier work, and that cache scales with the window size. On a 7B model, lifting n_ctx to 8,192 typically adds around 1GB to 2GB of VRAM, and it climbs further as the window grows. On a card with limited memory, that extra cache competes directly with the space the model weights need, so set the window to what your task genuinely requires rather than maxing it out by default.

This is the practical tension for anyone running models at home: model size versus window size, both fighting for the same VRAM. A machine built with that headroom in mind makes the trade-off far less painful. The AI PC range at Evetech is put together with exactly this kind of local-model work in view, and if you would rather start from what other SA buyers are running, the most popular PC builds right now are a useful reference point.

How to pick a sensible n_ctx

Match the window to the job. Short chatbot-style prompts run fine at 2,048 or 4,096. Summarising a long article or reasoning over a sizeable code file wants 8,192 or more. Watch your VRAM as you raise it: if the model spills out of the GPU into system RAM, generation slows dramatically. The sweet spot is the smallest window that comfortably fits your longest realistic prompt plus its answer.

Frequently Asked Questions

Does the context window include the model's reply?

Yes. The window is shared between your prompt and the generated output. A large prompt leaves less room for a long answer, so size n_ctx to cover both together.

What happens if my conversation exceeds the window?

The oldest tokens drop out of the window first. The model loses access to the earliest part of the chat, which is why long sessions on a small window start to feel forgetful.

How much VRAM does raising n_ctx use?

On a 7B model, going to 8,192 tokens adds roughly 1GB to 2GB of VRAM for the token cache. It grows with window size, so monitor memory as you increase it.

Is a bigger context window always better?

No. A larger window only helps if your task actually needs it, and it eats memory that could otherwise hold a larger or faster model. Pick the smallest window that fits your real workload.

Why does generation slow down when I set n_ctx too high?

If the larger cache pushes the model out of VRAM into system RAM, the GPU has to wait on slower memory. Keeping the whole session inside VRAM is what keeps generation fast.