Local Model OOMs at Long Context but Fits at Short? Blame the KV Cache

Your local model loads fine, you chat happily, then you paste a long document and it crashes with an out-of-memory error. The weights did not change, so what filled your VRAM? The KV cache did. It grows with every token of context, and at long context lengths it can quietly consume more memory than the model weights themselves.

Quick Answer

The KV cache scales linearly with context length, so a model that loads cleanly at 4K tokens can run out of VRAM at 32K because the cache balloons to fill your remaining headroom. The most direct fix is to lower the context window to what you actually need. Quantising the KV cache to 8-bit roughly halves that memory if you need the length back.

What The KV Cache Is And Why It Grows

Each time a model produces output, it builds up a running store of key and value vectors for every token already processed, avoiding redundant recomputation on each new step. That store is called the KV cache, and it makes generation fast. The catch is that it needs a fixed amount of memory per token, so its size scales directly with how many tokens are in context.

That is the trap. At 4K tokens the cache is small and sits comfortably alongside the weights. Push to 32K and you are holding eight times as much cache. A large model at a long context can burn many gigabytes on cache alone, which the simple "weights fit in VRAM" math never accounted for. So the model that loaded fine suddenly OOMs the moment you feed it a long prompt.

The Fastest Fix: Set The Context You Actually Need

Many runtimes default to a very large context window, and people leave it there without thinking. If your real workload is a few thousand tokens, reserving 128K is paying a huge VRAM tax for headroom you never use.

Drop the context setting to what the task genuinely requires. For a lot of chat and coding work, 4K to 8K is plenty, and that alone keeps the cache small enough to fit the model in VRAM without touching the model itself. This is the cleanest fix because it changes nothing about the model's quality, only the memory it reserves for context it was not going to use anyway. Choosing hardware with ample VRAM from the start avoids the squeeze entirely, and the AI PC range at Evetech stocks machines configured with the headroom to handle serious local-model workloads.

When You Genuinely Need The Length

If you really do need long context, you have options beyond just shrinking it. Quantising the KV cache to 8-bit cuts its memory footprint by roughly half, with only a slight effect on output quality, and some newer model families tolerate this especially well. Runtimes built around efficient cache handling also allocate memory on demand rather than reserving worst-case blocks, which reduces waste. And newer hybrid-attention model designs simply use far less cache at long context than older full-attention models, so choosing one of those can solve the problem at the architecture level. If more VRAM is the honest answer, the PC best sellers at Evetech reflect what local AI builders in SA are currently pairing with inference workloads.

Frequently Asked Questions

Why does the model load fine but crash on long prompts?

Because the weights are a fixed cost, but the KV cache grows with context length. A short prompt needs a small cache; a long one needs a much bigger one. When the cache plus the weights exceeds your VRAM, you hit an out-of-memory error even though loading the model alone worked.

Will reducing the context window hurt the model?

No, it does not change the model's capability, only how much past text it can hold at once. As long as your set context comfortably covers your actual prompts and replies, lowering it from an oversized default to a realistic value costs you nothing and frees real VRAM.

What does KV cache quantisation do to quality?

Compressing the cached keys and values to 8-bit roughly halves their memory use with only a slight quality impact. Some newer model families handle this particularly well. It is a good middle ground when you need a longer context than your VRAM would otherwise allow.

How do I know the KV cache is my problem and not the weights?

A telltale sign is a model that loads and runs at short context but fails only when the prompt gets long. If the failure is tied to context length rather than just loading the model, the cache is the cause. The weights would fail at load time, not partway through a long prompt.