32B Coding Model CUDA OOM on a 16GB GPU? Here Is the Fix

A 32B coding model at Q4 quantisation does not fit on a 16GB card, and CUDA tells you so the moment the loader tries to allocate the last few gigabytes of weights. The model expects roughly 20GB before the KV cache even appears, your GPU has 16GB, and the allocation fails. Nothing is broken. The maths simply does not add up, and once you see why, the fix is obvious.

Quick Answer

A 32B model at Q4_K_M weighs around 20GB in weights alone, then needs another 1GB to 2GB for the KV cache and runtime overhead, so it overflows a 16GB GPU before inference starts. The clean fixes are to drop to a smaller Q3 quant with a shorter context window, offload some layers to system RAM, or move to a 24GB card where the model fits with headroom.

Why 16GB cannot hold it

The rough rule for a GGUF model is parameters in billions multiplied by about 0.5 bytes at Q4, plus roughly 20 percent overhead. Run that for 32 billion parameters and the weights land near 20GB. Add the KV cache, which grows with your context length, and a realistic working figure for a 32B coding model at a useful context sits at 22GB to 24GB.

A 16GB card has no room for that. CUDA out of memory is not a bug or a driver fault, it is the allocator hitting the ceiling. The error often appears partway through loading because the first layers fit and the last ones do not.

The three real fixes

Drop the quant and the context. Moving from Q4_K_M to Q3_K_M shaves several gigabytes off the weights, and trimming the context window from 32k tokens down to 8k cuts the KV cache. Together that can squeeze a 32B model onto 16GB, at the cost of some answer quality and a shorter memory for long files. For quick code completions rather than whole-repository reasoning, that trade is often fine.

Offload layers to system RAM. Tools built on llama.cpp let you keep some layers on the GPU and push the rest to your CPU and RAM. The model then fits, but tokens slow down noticeably because part of the work crosses the slower path. It is a workable stopgap, not a daily driver for fast coding.

Get the VRAM the model actually wants. The honest fix is a 24GB GPU. At 24GB a 32B Q4 model fits with the headroom for a real context window and steady speed, with no offloading and no quality cut. If local coding models are part of your daily workflow, this is the upgrade that ends the problem rather than working around it.

Picking the card

For local LLM work, VRAM is the spec that decides what runs, well ahead of raw gaming performance. A 16GB card handles models up to roughly the 14B class at Q4 comfortably, while the 30B-plus coding models want 24GB. If you are pricing a build for this, the AI-ready PC range at Evetech is configured around exactly this GPU memory budget, and browsing the PC best sellers quickly shows which machines combine a strong CPU with a high-VRAM card.

Frequently Asked Questions

Why does the model load fine and then crash mid-prompt?

The weights may just fit, but the KV cache grows as your prompt and context fill. The crash arrives when that cache pushes total usage past 16GB. A shorter context window often stops it.

Will a smaller quant hurt my code quality?

Going from Q4 to Q3 trims accuracy slightly, and you may notice it on tricky multi-file reasoning. For routine completions and small functions the difference is usually minor, so test it on your own tasks before deciding.

Is CPU offloading worth it for coding?

It works and it gets the model running, but it is slow because part of every token crosses the CPU. It is fine for occasional use, frustrating as a full-time setup. A 24GB GPU keeps everything on the card.

How much VRAM do I need to run a 32B model properly?

Plan for 24GB to run a 32B model at Q4 with a real context window and steady speed. That is the tier where these models stop fighting you and just run.