Best gaming setup for running local LLMs in George

Quick Answer

A George setup for local LLMs should maximise GPU VRAM and pair it with ample system RAM. An RTX 4070 (12GB) runs mid-sized quantised models; an RTX 4080 (16GB) or 4090 (24GB) runs larger ones faster. Add 32GB to 64GB RAM and a fast NVMe SSD, with the tower around R28,000 to R55,000. Keep models fully in VRAM where possible, since overflowing into system RAM sharply slows responses. A 1TB or larger NVMe SSD keeps several large model files available with quick loading.

Choosing The Right GPU

The model size you can run depends on VRAM: 12GB suits quantised 7B to 13B models, 16GB extends the range, and 24GB handles larger models or higher precision with more headroom. The card's speed sets your tokens per second. NVIDIA is the practical choice because most LLM runtimes lean on its CUDA support.

Memory, Storage And Stability

System RAM acts as overflow when a model exceeds VRAM, so 32GB to 64GB adds flexibility for bigger models, though GPU-resident models run fastest. Model files are large, so a 1TB or larger NVMe SSD keeps several available with quick loading. A PSU sized to the GPU keeps inference stable under sustained use.

FAQ

Can I run a 70B model locally?

Large models like 70B need a lot of VRAM, often beyond a single consumer card, or heavy reliance on slower system RAM. Smaller quantised models are more practical at home.

How fast will responses be?

Speed depends on the GPU and model size, measured in tokens per second. A faster card with more VRAM gives quicker, smoother responses for a given model.

Is 16GB VRAM a good middle ground?

Yes, 16GB comfortably runs a wide range of quantised models and leaves headroom, making it a balanced choice between cost and capability for local LLMs.