Setting up a running local LLMs rig in George SA practical guide

Quick Answer

A local-LLM rig in George is sized by GPU VRAM: an RTX 4060 Ti 16 GB (~R10,500) runs quantised 7B-13B models like Llama 3 and Mistral well, while a 24 GB RTX 4090 handles 70B models. Pair it with a Ryzen 5 7600 and 32 GB system RAM. For local AI, VRAM capacity determines the model size you can run, far more than gaming-style frame rates.

Why VRAM is the key spec

Local LLMs load into GPU memory, so VRAM caps the model you can run fast:

• 12 GB: 7B-8B models (Mistral 7B, Llama 3 8B)
• 16 GB (RTX 4060 Ti 16 GB): 13B models, longer context windows
• 24 GB (RTX 4090): 70B quantised models

System RAM lets you offload extra layers to the CPU when a model just exceeds VRAM, though speed drops when it does, so size the GPU to your largest planned model.

A George local-LLM build

• GPU: RTX 4060 Ti 16 GB (~R10,500)
• CPU: Ryzen 5 7600 (~R4,200)
• RAM: 32 GB DDR5 (~R2,000)
• Storage: 1 TB NVMe (~R1,200) - model files are 4-40 GB each

George is on the Garden Route, so Evetech deliveries take 3-4 business days. A clean, stable power supply (650 W 80+ Gold) keeps long inference runs reliable; inspect and power-test before signing.

FAQ

What's the cheapest GPU that runs a 13B LLM?

The RTX 4060 Ti 16 GB (~R10,500) is the value pick for 13B models with Q4/Q5 quantisation. A 12 GB card handles 7B-8B models, but 16 GB gives the headroom for larger models and longer context.

How do I run an LLM locally?

Install Ollama or LM Studio, then download a quantised model (e.g. Llama 3 8B Q4). These tools manage the GPU offload automatically, generating tokens at usable speed on a 16 GB card.

Does Evetech deliver to George?

Yes, Evetech ships to George on the Garden Route in 3-4 business days. Power-test the GPU and confirm it's detected before signing the waybill.