Hardware for Running Local LLMs and Offline AI Coding: VRAM Guide

The single number that decides whether a local AI model runs usefully or crawls is VRAM. Everything else, the CPU, the system RAM, the storage, matters far less than how much memory sits on your graphics card. Pick a model that fits inside your GPU's VRAM and it answers in real time. Let even part of it spill into system RAM and your tokens-per-second collapses to a frustrating trickle. For an SA buyer budgeting in Rand, VRAM is the first spec to lock down, not the last.

Quick Answer

Size your GPU to the model. Size the GPU to the model you want to run. A 7B model quantised to Q4 sits comfortably inside 4 to 6GB of VRAM; a 32B wants around 20GB; a 70B demands 40 to 48GB. Stay inside that envelope and inference is fast; let the model spill into system RAM and speed falls to 2 to 3 tokens per second. VRAM first, everything else second.

Why VRAM Is the Hard Limit

A language model is a stack of weights that must be loaded into memory before the GPU can do anything with them. When all those weights fit in VRAM, the GPU streams through them at full bandwidth, which is where the speed comes from. The moment the model is larger than your VRAM, the runtime offloads the overflow layers to system RAM, and every token now has to wait while data shuffles across the much slower PCIe bus. That is why a model that "just barely doesn't fit" feels dramatically worse than one that fits with room to spare. There is no gentle slope here; it is a cliff.

Quantisation is the lever that moves a model up or down the VRAM ladder. Running a model at Q4 (four bits per weight) roughly halves its memory footprint compared to Q8, with a small and usually acceptable quality cost for coding and chat. So a 32B model that would need 40GB-plus at full precision drops to around 20GB at Q4. Knowing your quantisation is as important as knowing the parameter count. Browse the AI PC range at Evetech to line up VRAM against model size: picking the right pairing is the whole buying decision.

VRAM by Model Size: The Practical Bands

7B Models (4 to 6GB)

This is the entry point for offline coding assistants and chat. A 7B model at Q4 fits comfortably on a 8GB card with headroom for context. These run fast on mainstream GPUs and are the right starting point for most people experimenting with a local model. They handle autocomplete, short code generation, and Q and A well, though they lack the reasoning depth of larger models.

32B Models (around 20GB)

A 32B model is the sweet spot for serious offline AI coding. It reasons noticeably better than a 7B and handles larger, more complex prompts. The catch is the 20GB VRAM requirement at Q4, which pushes you into a 24GB-class card to leave room for context length. This is the tier where a lot of SA developers will want to land if they are doing real work offline.

70B Models (40 to 48GB)

The 70B class delivers the strongest reasoning but demands 40 to 48GB of VRAM, which a single consumer card cannot provide. You are looking at a 48GB workstation card or two cards working together, plus a power and cooling budget to match. For most individuals this is overkill; for a team running a shared inference box, it can make sense.

CPU, RAM and the Rest of the Build

None of this means the rest of the machine is irrelevant, only that it comes second. System RAM should comfortably exceed your model size so the OS and any overflow have room, with 32GB a sensible floor for serious work and 64GB if you intend to run large models partly in RAM. The CPU matters mainly for loading models and feeding the GPU; a mid-range modern chip is plenty, since inference itself is GPU-bound. Fast NVMe storage helps because large model files are tens of gigabytes each, and loading them from a slow drive turns every model switch into a wait. Spend your Rand on VRAM first, then RAM, then storage speed, and treat the CPU as the part you do not need to over-invest in.

Context Length Eats VRAM Too

Parameter count is not the whole memory story. The KV cache, the running memory of your conversation or code context, also lives in VRAM and grows with how long a context window you use. A long 32K-token context can add several gigabytes on top of the model weights. If you intend to feed an AI coding assistant entire files, budget extra VRAM beyond the raw model size, or you will hit the overflow cliff sooner than the model-size table suggests.

Spending Your Rand Wisely

The temptation is to chase parameter count, but a 32B model running fully in VRAM will out-perform a 70B model that spills into system RAM every single time. Buy the largest single block of VRAM your budget allows, prioritise it over clock speed or core count, and only then look at the rest of the build. System RAM still matters as a fallback and for the OS, but it is not where your model should live. To see which configurations local buyers are actually choosing, the best-selling PC list at Evetech is a useful sanity check against your shortlist.

Frequently Asked Questions

How much VRAM do I need to run a local LLM?

It depends on the model. A 7B model at Q4 needs 4 to 6GB, a 32B needs around 20GB, and a 70B needs 40 to 48GB. Always size your GPU to the model plus extra for context, not the other way around.

What happens if a model is bigger than my VRAM?

The runtime offloads the overflow into system RAM, and every token then waits on the slow PCIe bus. Speed collapses to roughly 2 to 3 tokens per second, which makes the model effectively unusable for interactive work.

Does quantisation reduce VRAM use?

Yes, significantly. Running at Q4 instead of full precision roughly halves the memory footprint with a small quality trade-off. It is the main reason a 32B model can fit in around 20GB instead of 40GB-plus.

Is system RAM a substitute for VRAM?

No. System RAM is far slower for inference and only acts as an overflow fallback. Relying on it kills your tokens-per-second. Treat VRAM as the real budget and system RAM as a safety margin.

Can a single consumer GPU run a 70B model?

Not at usable speed. A 70B model needs 40 to 48GB, which exceeds a single consumer card. You need a 48GB workstation card or a multi-GPU setup, along with the power and cooling to support it.