VRAM Rule of Thumb: Calculate What Model Your GPU Can Run

Before you spend on a graphics card to run AI models at home, one number decides almost everything: how much dedicated video memory it carries. The VRAM rule of thumb turns a confusing spec sheet into a simple ceiling, telling you the largest local language model a given GPU can actually load and run at the popular Q4 quantisation. Get this right and you avoid buying a card that chokes on the model you wanted.

Quick Answer

At Q4 quantisation, a 7B model needs roughly 4GB to 6GB of VRAM, a 13B needs about 8GB to 10GB, a 32B needs around 20GB to 24GB, and a 70B needs 40GB or more. Match the model tier to the card's memory and you know its ceiling before you buy.

How the rule actually works

Q4 quantisation compresses a model's weights to 4-bit precision, cutting memory use by roughly three quarters versus full 16-bit precision while keeping output quality close to the original. Q4_K_M, the most common variant, is widely treated as the community sweet spot because it stays close to full quality at a fraction of the size.

The maths is approximate but reliable. Take the parameter count in billions, halve it for a rough 4-bit byte count, then add overhead for the context window and the operating system's own memory needs. That overhead is why a model that looks like it should fit in exactly 8GB often will not run comfortably on an 8GB card.

The bands in practice

A 7B model is the entry point. At Q4 it lands around 4GB to 6GB, so even a modest 8GB card handles it with room to spare for context. A 13B model needs roughly 8GB to 10GB, which means an 8GB card can technically load it but leaves almost no headroom, so a 12GB card is the comfortable target.

A 32B model is where things get serious, needing about 20GB to 24GB. That points at a 24GB card such as a high-end consumer GPU. A 70B model is the heavyweight at 40GB or more, which usually means either a workstation-class card or two 24GB cards running together. Knowing these bands up front stops you from buying a card one tier short of the model you actually wanted to run. The current AI-capable desktops sit in the AI PC range, and the cards that move fastest among local-AI buyers tend to be the higher-VRAM models in the most popular PC picks.

Frequently Asked Questions

Why does a 13B model not fit comfortably on an 8GB card?

The weights alone sit near the 8GB line, but the context window and system overhead push the real requirement past it. A 12GB card gives the headroom to load the model and hold a usable conversation length.

Does a higher quantisation than Q4 change these numbers?

Yes. Q8 roughly doubles the memory versus Q4, while smaller quants like Q3 shrink it further at some quality cost. Q4_K_M is the popular balance because it stays close to full quality while keeping memory low.

Can I run a model that exceeds my VRAM?

You can, by offloading layers to system RAM, but speed drops sharply because system memory is far slower than VRAM. The rule of thumb assumes the model fits entirely in video memory for usable performance.

Is VRAM the only spec that matters?

It is the hard ceiling on which models will load at all, so it matters most. Once a model fits, GPU compute and memory bandwidth decide how fast it generates text.