Tokens/Sec Tanking on Long Files? Trim the KV Cache Context

You load a short prompt into your local coding model and it flies. You feed it a 2,000-line file and the same model crawls, spitting tokens out at a fraction of the speed. The model did not get slower. Your KV cache quietly overflowed your VRAM and pushed part of the workload onto system RAM. Capping your context window is usually the one-setting fix that brings the speed straight back.

Quick Answer

The KV cache grows with every token of context, eating VRAM as it goes. Once the model plus its cache exceed your GPU memory, inference spills to system RAM over PCIe and throughput can collapse by 5x to 15x. Cap the context window so the whole cache stays in VRAM and you keep full token-per-second speed. On a 12GB card, a 7B model is comfortable around 16K to 32K context.

What the KV Cache Actually Is

During inference, the model caches the attention keys and values for each token already processed, so it can skip recomputing them on the next step. That accumulating store is the KV cache, and it scales linearly: every token of your prompt and every token the model produces adds to it. A long code file is thousands of tokens, so the cache balloons as the context fills.

This is separate from the memory the model weights occupy. The weights are a fixed cost loaded once. The cache is a running cost that climbs the longer your conversation or file gets, which is exactly why a session that started fast can grind to a halt later.

Why It Tanks Your Speed

A GPU is fast because it works out of its own VRAM. The moment your weights plus your KV cache no longer fit, the system offloads the overflow to ordinary system RAM, and the GPU has to wait on data crossing the PCIe bus to keep going. That wait is the bottleneck. Tokens per second do not dip gently; they fall off a cliff, often dropping to a small fraction of the in-VRAM rate.

The rough rule on consumer hardware: you can usually fit a big model or a long context, rarely both at full speed. A larger 14B model may already be tight at 16K context on a 12GB card and spill at 32K, while a 7B model has more headroom.

How to Fix Throughput Without New Hardware

The fastest fix costs nothing. In your runner (LM Studio, llama.cpp and similar), set a context length your VRAM can actually hold rather than leaving it at the maximum. If a long file no longer fits, the model will not silently offload it; you keep the speed and trim what you feed in. The right ceiling depends on your card and model size, so nudge it down until tokens per second stay high. A capable AI-focused machine like those in the AI PCs range at Evetech gives you the VRAM headroom to push that ceiling higher in the first place.

A second lever is KV cache quantisation. Storing the cache at 8-bit roughly halves the memory it consumes per token, letting you keep a longer context in VRAM at a small cost to output quality. Combine a sensible context cap with quantised cache and you fit far more context before anything spills. If your work genuinely needs huge files in one pass, more VRAM is the real answer, and the PC best sellers at Evetech show which GPU and memory configs local-AI builders are choosing.

Frequently Asked Questions

Why does my local model slow down only on long files?

Because the KV cache grows with every token. A long file fills the cache until it no longer fits in VRAM, at which point the overflow spills to system RAM and the GPU stalls waiting on PCIe transfers, collapsing your token rate.

How much VRAM does the KV cache use?

Roughly 0.11GB per 1,000 tokens for a 7B model, and far more at scale. An 8B model at full precision can need around 4GB of cache at 32K context and about 16GB at 128K, on top of the weights themselves.

Will capping context lose information from my file?

It limits how much the model sees at once, so you trade total context for sustained speed. For most coding tasks a sensible cap keeps the relevant window in view while avoiding the offload that destroys throughput.

Does KV cache quantisation hurt quality?

Storing the cache at 8-bit roughly halves its memory cost with only a slight degradation in output. For most local coding work that trade is well worth the extra context you can keep in VRAM.