How to Avoid VRAM-Overflow Slowdowns When Running Local Models

A local model that ran fast yesterday and crawls today usually has one cause: it no longer fits in your GPU's memory. The moment a model spills past your VRAM budget, layers fall back to system RAM and inference speed collapses, often from dozens of tokens a second to a painful trickle. Avoiding that slowdown is mostly about sizing the model and its quantisation to land inside the card before you ever hit run.

Quick Answer

Pick a model and quantisation level that fit inside your VRAM with headroom for the context cache. As a rough guide at Q4_K_M with a modest context, a 7 to 8B model needs about 6 to 7GB, a 13 to 14B about 10 to 12GB, a 32B about 21 to 23GB, and a 70B around 43 to 46GB. Leave 30 to 50 percent of headroom beyond the raw model size for the KV cache, and the model stays on the GPU.

Step 1: Know your real VRAM budget

Open your GPU's stats and note the total VRAM, then subtract what the desktop and other apps already use. An 8GB card does not give you 8GB for the model; the OS, browser and display take a slice first. Plan around what is actually free, not the sticker number.

This is why a card's VRAM matters more than its raw speed for local inference. A slower card with more memory will run a model that a faster card with less memory simply cannot hold. The AI and workstation PC range at Evetech is built around exactly this: GPUs chosen for memory capacity, not just frame rates.

Step 2: Choose the quantisation that fits

Quantisation shrinks a model by storing its weights at fewer bits. Lower precision means a smaller file and less VRAM used, at a small cost in quality.

For most people the sweet spot is Q4_K_M, roughly 4.85 bits per weight. It is about 70 percent smaller than the full-precision FP16 version with quality loss that is hard to notice in everyday use. If you have memory to spare, Q5_K_M or Q6 give a touch more quality. If you are tight, Q3 variants squeeze further but the quality drop starts to show. Match the quant to your card: do not load an FP16 model on an 8GB GPU and wonder why it crawls.

Step 3: Match the parameter count to your card

Use the rough fit table as your starting point:

1. 4GB VRAM: stick to sub-2B models.
2. 6GB VRAM: 3B to 3.8B models are comfortable.
3. 8GB VRAM: a 7B to 8B model at Q4_K_M fits well.
4. 12GB VRAM: 13B to 14B becomes practical.
5. 24GB VRAM: 32B class models are in reach.

If a model two sizes above your card keeps overflowing, that is the table telling you to drop down a tier rather than fight it.

Step 4: Control the context window

This is the trap people miss. The KV cache, the model's working memory for the conversation, grows linearly with context length and lives in VRAM alongside the weights. A model that fits at 2,048 tokens of context can overflow at 32,000.

So if you are close to the limit, cut the context window. Dropping to 2,048 tokens frees 1 to 2GB. Only raise it when you genuinely need long inputs. Modern architectures using Grouped Query Attention (as in recent Llama, Mistral and Qwen models) already use the cache more efficiently, which is one reason to prefer current models over older ones.

Step 5: Use partial offload when you must

If a model is slightly too big and you refuse to drop a size, your tools can split it. In Ollama the setting is num_gpu; in llama.cpp it is the -ngl flag. You put as many layers as fit on the GPU and the rest on the CPU.

Be honest about the cost: any layer running on the CPU is far slower, so partial offload trades speed for the ability to run at all. It is a fallback for huge models or long contexts on limited hardware, not the default you should aim for. The goal is always to fit the whole thing in VRAM; offload is the compromise when you cannot.

Step 6: Verify it actually fit

After loading, check VRAM usage against your card's total. If you are pinned at the limit, you have no headroom for the cache to grow during a long chat and you will hit a slowdown or an out-of-memory crash mid-conversation. Drop a quant level or shorten the context until there is breathing room. The current PC best sellers are a useful reference for the cards SA builders are pairing with local AI workloads right now.

Frequently Asked Questions

What is VRAM overflow exactly?

It is when a model plus its context cache needs more memory than the GPU has, so part of it falls back to system RAM. CPU-bound layers run far slower than GPU layers, so inference speed drops sharply. Keeping everything in VRAM is what keeps it fast.

Which quantisation should I start with?

Q4_K_M for most cases. It is roughly 70 percent smaller than full precision with quality loss most users never notice. Go higher (Q5, Q6) if you have spare VRAM, lower (Q3) only if you must squeeze a model onto a small card.

Why does my model slow down only in long chats?

The KV cache grows with the length of the conversation and consumes VRAM as it goes. A model that fit at the start can overflow once the context fills up. Shorten the context window or leave more headroom at load time.

Is more VRAM better than a faster GPU for local AI?

For fitting larger models, yes. A card with more memory can hold a model that a faster but smaller card cannot run at all. Capacity sets the ceiling on model size; speed only affects how fast that model runs once it fits.

What does GPU layer offloading do?

It splits a model so some layers run on the GPU and the rest on the CPU, letting you run a model too big to fit entirely in VRAM. It works, but the CPU layers are slow, so use it only when fitting the whole model is not possible.