How to Choose a Quantisation Level for a Local Coding Model

The single decision that makes or breaks a local coding assistant is the quantisation level you load it at, because it decides whether the model fits entirely in your graphics card memory or spills over into far slower system RAM. Pick too high a precision and the model crawls; pick too low and the code suggestions get sloppy. The sweet spot for most setups is narrower than people expect, and it depends almost entirely on how much VRAM you have to work with.

Quick Answer

For a 7B or 8B coding model on an 8GB card, run Q4_K_M, which keeps the whole model in VRAM and preserves roughly 95 to 98 percent of full-precision quality. If you have 16GB or more, step up to Q5_K_M or Q8_0 for cleaner output. Q8 is effectively lossless but doubles the memory footprint of Q4.

What the Q numbers actually mean

The Q4, Q5, and Q8 labels describe the average number of bits used to store each weight in the model. Full precision is 16 bits per weight, so Q8 roughly halves the file size, Q5 cuts it further, and Q4 quarters it. The _K_M suffix matters too. It marks the modern k-quant method used by llama.cpp, which is noticeably sharper than the old Q4_0 scheme at the same bit count because it spends more bits on the weights that matter most. When you download a model in GGUF format, the file name carries this label, and that single tag tells you both how much memory it will eat and how close to the original it will behave.

Matching the level to your VRAM

The whole exercise is about fitting the model plus its working context inside the card. A 7B model at Q4_K_M lands around 4 to 5GB on disk and loads comfortably into 8GB of VRAM with room left for context. The same model at Q8_0 needs roughly 16GB, so on an 8GB card it overflows and the slow part of the load runs on the CPU, dropping you from instant responses to a frustrating wait per reply. The practical rule is simple. Find the largest level that leaves a couple of gigabytes of VRAM free after the model and your typical context window are loaded, and stop there. More than enough headroom is wasted; too little and you have spilled into system RAM without realising it.

Why coding tasks deserve a higher level

General chat tolerates aggressive quantisation because a slightly fuzzy word choice rarely matters. Code is less forgiving. A single transposed token can produce a function that looks plausible and fails to compile, or worse, compiles and behaves wrongly. Benchmarks consistently show Q4 k-quants holding most of their quality, but the gap to Q5 and Q8 shows up exactly in the precise, structured output that programming demands. So if your hardware can carry it, lean one level higher for coding than you would for casual use. On a card with the memory to spare, Q5_K_M is a reasonable default for code, and Q8_0 is worth it when correctness is the priority. For anyone weighing a hardware upgrade to run bigger models, the AI ready PC range groups the machines built with the VRAM these workloads actually need.

A quick decision path

Start from your graphics card memory and work down. With 8GB, Q4_K_M of a 7B to 8B model is the dependable choice. With 12GB, you can move to Q5_K_M or fit a larger model at Q4. With 16GB or 24GB, Q8_0 of a mid-size model or Q4 to Q5 of a larger one becomes realistic, and that is where local coding starts to feel genuinely useful rather than a novelty. The tools that load these files, such as Ollama and llama.cpp, read the GGUF tag automatically, so once you have chosen the level the rest is a download.

Frequently Asked Questions

Is Q4 good enough for serious coding work?

For most people, yes. Q4_K_M keeps around 95 to 98 percent of the original model's quality and runs fast on modest hardware. If your card has spare memory, Q5_K_M gives a small but real accuracy bump that coding tasks benefit from.

What happens if the model does not fit in VRAM?

The overflow loads into system RAM and runs on the CPU, which is dramatically slower. Responses that should take a second can take many times longer. This is the exact problem choosing the right quantisation level avoids.

Does Q8 noticeably beat Q4 for code?

Q8_0 is close to lossless and edges out Q4 on precision-sensitive output like code, but it needs roughly double the memory. The gain is real yet modest, so only reach for it when you have the VRAM to spare.

What is GGUF and do I need it?

GGUF is the standard file container for quantised models, supported directly by Ollama and llama.cpp. Almost every locally runnable coding model is published in GGUF, and the quantisation level is baked into the file name.

How much VRAM do I need for a 7B coding model?

Around 8GB comfortably runs a 7B model at Q4_K_M with usable context. Stepping up to Q5 or Q8 of the same model pushes you toward 12GB and 16GB respectively.