What Is Quantisation? How It Shrinks AI Models to Fit Smaller GPUs

Running a capable AI model on a GPU you can actually afford usually comes down to one trick: quantisation. It stores a model's weights at lower numerical precision, say 4-bit integers instead of 16-bit floating point numbers, so the same model occupies a fraction of the memory it used to. The result is that a model that would never load on your card suddenly fits, with only a modest hit to quality.

Quick Answer

Quantisation shrinks an AI model by storing its weights at lower precision. A 7B-parameter model that needs about 14GB of VRAM at FP16 drops to roughly 4.4GB at Q4_K_M, a 4-bit format. That means a model demanding a high-end card can run comfortably on a mid-range GPU with 8GB of memory.

What precision actually means

Every weight in a neural network is a number, and the format used to store that number decides how much memory it takes. FP16 uses 16 bits per weight, giving high accuracy at a high memory cost. Quantisation rounds those numbers into a smaller format, commonly 8-bit or 4-bit integers, so each weight takes a quarter or an eighth of the space.

The headline maths is simple. Multiply parameter count by bytes per weight. A 7B model at FP16 (2 bytes per weight) is about 14GB. The same model at Q4_K_M (roughly half a byte per weight after overhead) lands near 4.4GB. That difference is what moves a model from "needs a workstation card" to "runs on a desktop GPU you already own." If you want a machine already specced for this kind of work, the AI PCs range at Evetech is a sensible starting point.

The quality trade-off, honestly

Lower precision means some information is lost. The good news is that modern quantisation formats are clever about where they cut. Schemes like Q4_K_M keep the most sensitive layers at higher precision and squeeze the rest, so the quality drop is small enough that most users cannot tell the difference in everyday chat or coding tasks.

The general rule: 8-bit quantisation is nearly indistinguishable from full precision. 4-bit is the sweet spot most people run, balancing size against quality. Below 4-bit, say 3-bit or 2-bit, the model starts making more mistakes, and you only go there when memory is desperately tight.

GGUF and why the format names matter

If you have downloaded a local model, you have probably seen GGUF files with labels like Q4_K_M, Q5_K_S, or Q8_0. GGUF is the container format used by popular local inference tools. The suffix tells you the precision: the number is the bit depth, and the letters indicate the variant and how aggressively certain layers are compressed. Q4_K_M is the most widely recommended balance for consumer cards.

Picking a quantisation for your VRAM

Match the model file to the memory on your card with a little headroom for context. As a rough guide:

• 8GB GPU: a 7B or 8B model at Q4_K_M fits comfortably.
• 12GB GPU: you can run larger 13B models at 4-bit, or 7B at higher precision.
• 16GB and up: 13B at Q5 or Q6, or you start reaching toward larger models.

Leave a couple of gigabytes free for the context window, which grows with longer prompts. To see which cards builders are pairing with local AI work right now, the best-selling PC list is a quick reference point.

Frequently Asked Questions

Does quantisation make a model dumber?

Slightly, but far less than the size saving suggests. At 8-bit the difference is negligible, and at 4-bit most users notice nothing in normal use. Only very aggressive 2-bit and 3-bit formats produce a noticeable drop in accuracy.

What is Q4_K_M and why is it recommended?

Q4_K_M is a 4-bit GGUF quantisation that keeps the most important layers at higher precision while compressing the rest. It is the most popular choice because it offers the best balance of small file size and retained quality for consumer GPUs.

How do I know what model size my GPU can run?

Take the model's parameter count and multiply by the bytes per weight of the quantisation, then add a couple of gigabytes for context. A 7B model at Q4_K_M needs about 4.4GB plus overhead, which fits an 8GB card easily.

Can I run quantised models without a powerful GPU?

Yes. The whole point of quantisation is to lower the hardware bar. A mid-range GPU with 8GB of VRAM can run a quantised 7B model that would otherwise demand far more expensive hardware.