CUDA Out of Memory Running a 70B Model? Why You Need 40-48GB VRAM

A single 24GB card loading a 70B model will throw CUDA out of memory before it finishes reading the weights, every time. The maths is unforgiving: at Q4_K_M quantisation, a 70-billion-parameter model wants roughly 40 to 48GB of VRAM once the KV cache and runtime overhead are added on top of the weights. Twenty-four gigabytes was never going to hold it. The fix is not a magic flag, it is choosing one of three real paths.

Quick Answer

At Q4_K_M quantisation, a 70B model demands around 40 to 48GB of VRAM, which is well beyond a single 24GB card's capacity. Your three options are: drop to a smaller quant or a smaller model, split the load across two 24GB GPUs for 48GB combined, or offload some layers to system RAM and accept much slower speed. There is no setting that shrinks a 70B to fit 24GB without one of these trade-offs.

Where the 40 to 48GB number comes from

The weights are the big fixed cost. At Q4_K_M a 70B model is roughly 38 to 42GB of weights alone. On top of that sits the KV cache, which grows with context length, plus a smaller fixed tax for activations and the CUDA runtime, usually another 15 to 20 percent. Add it up and 48GB is the realistic single-card entry ticket for a 70B at this quant with a usable context window. Try to fit it into exactly 40GB with a long context and you will OOM the moment the cache fills.

Option 1: Two GPUs for 48GB combined

Two 24GB cards give you 48GB of pooled VRAM, which clears the bar for a 70B at Q4_K_M. Ollama splits a model across multiple GPUs automatically, and llama.cpp does it with the tensor-split argument. This is the cleanest path to full-speed 70B inference on consumer-class hardware, and the AI PC range at Evetech includes dual-GPU and high-VRAM options spec'd specifically for this workload.

Option 2: Offload layers to system RAM

If you only have one card, offload the layers that will not fit onto CPU and RAM. In llama.cpp you cap the GPU layers so the remainder runs on the processor. A 70B has 80 layers, and with 24GB you can push roughly 45 to 50 of them onto the GPU while the rest sit in RAM. It works, and it will not OOM, but inference slows dramatically because the CPU-bound layers become the bottleneck. Fine for occasional queries, painful for interactive coding.

Option 3: Drop the quant or the model

You can also shrink the demand. A heavier quantisation like Q3 or Q2 reduces the weight footprint, though quality degrades as you go lower. Alternatively, run a smaller model: a 32B or 34B at Q4 fits comfortably on a single 24GB card and is often the smarter pick for a one-GPU rig. Pick a quant that leaves one to two gigabytes of headroom below your card's VRAM, never one that exactly equals it.

A quick lever that helps in all three cases: enable KV cache quantisation. Newer llama.cpp builds support Q8 KV cache, which roughly halves the cache overhead and buys back useful headroom. If you want to see what a well-specced local-LLM machine looks like in practice, the PC best sellers at Evetech show which high-VRAM configurations SA buyers are actually running.

Frequently Asked Questions

Can I run a 70B model on a single 24GB GPU?

Not fully on the GPU. The weights and cache need 40 to 48GB, so a 24GB card must offload layers to system RAM, which works but slows inference heavily. For full-speed 70B you need around 48GB, typically two 24GB cards.

How much VRAM does a 70B model need at Q4_K_M?

About 40 to 48GB. The weights are roughly 38 to 42GB, and the KV cache plus runtime overhead adds another 15 to 20 percent. Aim for 48GB to leave headroom for a reasonable context length.

Will a smaller quantisation fix the OOM error?

It reduces the footprint, so Q3 or Q2 may fit a 70B onto less VRAM, but quality drops as you quantise harder. Often the better move is running a 32B or 34B model at Q4, which fits a single 24GB card cleanly.

Does enabling KV cache quantisation help?

Yes. Q8 KV cache in recent llama.cpp builds roughly halves the cache overhead, freeing VRAM for weights or a longer context. It is one of the easiest wins when you are sitting just over your card's limit.