RTX 4090 OOM on a 70B Model? Why 24GB Falls Short

Load a 70B model in Q4_K_M onto an RTX 4090 and the run dies before it starts, with a CUDA out-of-memory error every time. The arithmetic is brutal and unforgiving: the weights alone want more memory than the card physically has, so no amount of tweaking batch size or context length rescues it. The good news is that the 4090 is still a serious local-AI card once you size your model to its 24GB.

Quick Answer

Running a 70B model at Q4_K_M requires roughly 40 to 48GB of VRAM once context is factored in, which a single 24GB RTX 4090 cannot hold, so OOM is guaranteed. Drop to a 32B model at Q4_K_M, which lands around 19 to 20GB, and the 4090 runs it comfortably and fast.

The VRAM Maths Behind the OOM

Quantisation shrinks a model but does not work miracles. At Q4_K_M, a 70B model's weights occupy somewhere in the region of 40 to 42GB on their own. Add the key-value cache for any usable context window plus the runtime's overhead, and the real requirement climbs toward 48GB. The RTX 4090 ships with 24GB of GDDR6X. There is no setting that fits 42GB of weights into 24GB of memory, which is why the failure is immediate and absolute rather than a slowdown.

This is also why two RTX 4090s, giving 48GB combined, is the usual consumer route to running 70B locally at Q4_K_M with room for context.

What the 4090 Runs Well

Size the model to the card and the 4090 is excellent. A 32B model at Q4_K_M sits around 19 to 20GB, leaving headroom for a healthy context window, and the 4090 generates at high speed because the entire model lives in fast VRAM with nothing spilling to system memory. That sweet spot, models at or below 32B, is where a single 4090 genuinely shines for local inference. If you want to compare complete AI-ready machines rather than building from parts, the AI PC range at Evetech lists current options by spec.

If You Must Run 70B

Two realistic paths exist, both with a cost:

• CPU offload. Push the layers that do not fit into system RAM. It works, but generation speed collapses, often to single-digit tokens per second, because the CPU and its slower memory now sit in the critical path.
• Two-card setup. A pair of 24GB cards splits the model across 48GB of combined VRAM. This keeps everything on GPUs but introduces PCIe bandwidth as a limiting factor, so it is faster than offload yet still below the snappy feel of a model that fits one card.

For most South African users the cleaner answer is to run a 32B model that fits natively rather than fight a 70B that does not. Browsing the PC best sellers at Evetech gives a quick read on which configurations local buyers have settled on for this kind of work.

Frequently Asked Questions

Why does the 4090 OOM on a 70B model but not a 32B one?

A 70B model at Q4_K_M needs around 40 to 48GB once context is counted, well past the card's 24GB. A 32B model at the same quantisation lands near 19 to 20GB, which fits with headroom to spare.

Can a lower quantisation make a 70B fit on 24GB?

Not at any quality worth running. Even aggressive sub-4-bit quantisation leaves the weights too large for 24GB with usable context, and the quality loss makes it a poor trade. Stepping down to a 32B model is the better move.

Will two RTX 4090s run a 70B model?

Yes. Two 24GB cards give 48GB combined, enough for a 70B at Q4_K_M with context. The limit becomes PCIe bandwidth between cards rather than memory capacity.

What is the largest model a single 4090 runs comfortably?

A 32B model at Q4_K_M is the practical ceiling, sitting around 19 to 20GB and leaving room for context. That keeps everything in VRAM for full-speed generation.