Quick Answer

A Knysna setup for local LLMs centres on GPU VRAM, with system RAM as backup. An RTX 4070 (12GB) runs quantised mid-sized models; stepping to 16GB or 24GB runs larger ones faster. Pair with 32GB to 64GB RAM and a 1TB NVMe SSD, with the tower around R28,000 to R55,000. Quantisation lets capable models fit in limited VRAM, so it is key to running strong models at home. A PSU matched to the GPU's draw keeps inference stable under sustained use.

How Much Model Fits

VRAM is the gatekeeper for local language models. A 12GB card runs quantised 7B to 13B models smoothly, which covers many strong open models. More VRAM lets you run larger or higher-precision models entirely on the GPU, which is far faster than offloading to system RAM. NVIDIA cards suit the CUDA-based tooling most runtimes use.

Keeping Inference Smooth

System RAM of 32GB to 64GB helps when models spill beyond VRAM, though that runs slower. A 1TB or larger NVMe SSD stores several large model files with quick loading. The PSU should match the GPU's draw for stable operation. The CPU plays a minor role, so a capable mid-range chip is enough for this GPU-led task.

FAQ

What does quantisation do for local LLMs?

It compresses a model so it uses less memory, letting larger models fit in limited VRAM with minimal quality loss. It is key to running capable models at home.

Will offloading to system RAM work?

Yes, but it is slower than running entirely in VRAM. It lets you load bigger models than the GPU alone allows, at a cost in response speed.

How big are local LLM model files?

They range from a few gigabytes to tens of gigabytes depending on size and quantisation. A 1TB SSD comfortably holds several, with more for larger collections.

TIP

Pro Tip

Keep models fully within VRAM whenever possible. The moment a model overflows into system RAM, response speed drops sharply, so picking a model size that fits your card matters as much as the card itself.