A model that throws a CUDA out-of-memory error the moment you load it is not a dead end, it is usually a sign you are running it at the wrong precision. Q4_K_M quantisation is the first thing to reach for: it stores the model's weights in 4-bit form instead of 16-bit, cutting the memory footprint by roughly 75 percent while keeping almost all of the original quality. A model that refused to fit in your VRAM at FP16 often loads cleanly with headroom to spare once it is quantised.

Quick Answer

Q4_K_M shrinks a model to about a quarter of its FP16 size while retaining around 97 to 99 percent of measured quality, so the practical loss is under three percent and often imperceptible in chat and coding tasks. A 7B model drops from roughly 14 GB at FP16 to about 4 to 4.5 GB at Q4_K_M, which is why it fits comfortably on 6 to 16 GB consumer cards.

What Q4_K_M actually does

The name encodes the recipe. The "Q4" means most weights are stored at 4 bits instead of 16. The "K" means it uses K-quants, a mixed-precision scheme rather than a flat conversion. The "M" stands for medium, the balanced middle option between the smaller "S" and larger "L" variants.

The clever part is that Q4_K_M does not treat every layer the same. It keeps the most sensitive tensors, the attention output and value projections, at a higher 6-bit precision and quantises the rest at 4-bit. That selective approach is why it beats plain Q4_0 on quality at almost the same size, and it is the reason the local-model community treats Q4_K_M as the default sweet spot.

How much VRAM you actually save

The maths is easy to reason about. At FP16, every billion parameters needs roughly 2 GB just for the weights. At Q4_K_M that drops to a little over half a gigabyte per billion. So a 7B model goes from about 14 GB to around 4 to 4.5 GB, a 13B from about 26 GB to roughly 8 GB, and a 70B that was hopeless on a single card becomes a serious option on a 48 GB setup or two 24 GB cards.

Remember to leave room on top of the weights for the context window and key-value cache, which grow with how much text the model holds in memory at once. A good rule of thumb is to size your card so the quantised weights take no more than about 80 percent of your VRAM, leaving the rest for context.

When to step up or down

Q4_K_M is the right starting point for most people, but it is not the only option. If you have spare VRAM and want the last sliver of accuracy, Q5_K_M or Q6_K give you a little more quality for a modest size increase. If you are still just barely over the line, Q4_K_S or the more aggressive 3-bit quants will squeeze in, though quality starts to slip noticeably below 4-bit. The honest answer is to run Q4_K_M first and only move if you have a measured reason to.

If your card is the real bottleneck rather than your settings, it may be time to look at a GPU with more memory. Machines in the AI PC range at Evetech are configured with the VRAM headroom to handle larger models comfortably. The PC best sellers list at Evetech is a quick gauge of which complete systems local-model builders in South Africa are actually buying right now.

Frequently Asked Questions

Will I notice the quality drop from Q4_K_M?

For chat, summarisation and most coding, almost never. Benchmarks put the loss at roughly one to three percent against FP16, and in everyday use the output reads the same. Highly precise tasks like long-chain mathematical reasoning are where the gap, if any, shows up first.

Does quantisation make the model run faster?

Often yes, because there are fewer bits to move between memory and the compute cores, and the model now fits entirely in fast VRAM instead of spilling to system RAM. The speed gain varies by hardware, but escaping an out-of-memory spill is usually the bigger win.

Where do I get a Q4_K_M version of a model?

Most popular open models are published in GGUF format with Q4_K_M already provided, so you download that file directly rather than converting it yourself. If only the full-precision weights exist, llama.cpp ships the tools to quantise them locally.

How much VRAM do I need for a 7B model at Q4_K_M?

Plan for roughly 5 to 6 GB total: about 4 to 4.5 GB for the weights plus a margin for context. That puts 7B models comfortably within reach of an 8 GB card, which is why entry-level RTX hardware is enough to start.

If a bigger model is what you are really after, the fix may be more VRAM rather than more compression. Compare local-AI ready machines on the PC best sellers page at Evetech and match the card to the model you want to run.