How to Pick a Local AI Model Your PC Can Actually Handle

The fastest way to waste an afternoon with local AI is downloading a model that is too big for your graphics card and watching it crawl. There is a simple rule that avoids it: the model file has to fit inside your VRAM with a little room to spare. Get that match right and the model runs quickly on your own machine, no cloud subscription needed.

Quick Answer

Pick a model whose GGUF file size fits inside your GPU VRAM with 1GB to 2GB of headroom left for context. An 8GB graphics card comfortably runs a 7B model at Q4 quantisation but will struggle with a 13B one. Match the file size to your VRAM first, everything else is secondary.

The VRAM Rule That Decides Everything

Local models run fastest when they sit entirely in your graphics card's VRAM. If the model is bigger than your VRAM, part of it spills into slower system RAM and the whole thing drags. So the first check is always: how much VRAM do I have, and does the model file fit inside it?

Leave 1GB to 2GB free on top of the model size. That headroom holds the context window, which is the conversation or document the model is working with. Fill the VRAM right to the brim and you will run out of room the moment you give it a long prompt.

Reading GGUF Sizes and Quantisation

Most local models are distributed as GGUF files, and the filename usually tells you the quantisation level, written as something like Q4, Q5 or Q8. Quantisation shrinks the model by storing its weights at lower precision. Q4 is the popular sweet spot, roughly quartering the size of the full model while keeping output quality high.

Lower quantisation means a smaller file and lower quality, higher means a bigger file and better quality. For most people, Q4 of a larger model beats Q8 of a smaller one, because the bigger model has more capability to begin with. Check the actual file size on the download page rather than guessing from the parameter count, since quantisation changes the footprint dramatically.

Matching Cards to Models

Here is the practical pairing. An 8GB card runs 7B models at Q4 well, which covers chat, writing help and code completion. A 12GB card opens up 13B models comfortably and adds context room. A 16GB card handles larger models or gives a 13B plenty of breathing space, and 24GB cards run into the 30B-class territory.

The honest reading is that VRAM, not raw gaming performance, is what limits local AI. A card with more memory but fewer gaming frames may run bigger models than a faster card with less VRAM. If you are choosing hardware specifically for this, the AI-ready PC range at Evetech groups systems by the kind of memory that local models actually care about. For a broader comparison of popular builds, the PC best sellers show what SA buyers are currently pairing with local AI work.

Frequently Asked Questions

What model size fits an 8GB graphics card?

A 7B model at Q4 quantisation fits an 8GB card comfortably, leaving room for context. A 13B model is too large and will spill into system RAM, slowing it down badly.

What does Q4 mean in a model name?

Q4 is the quantisation level, meaning the weights are stored at roughly 4-bit precision. It shrinks the file to about a quarter of full size while keeping quality high, which is why it is the most common choice.

Should I pick a bigger model at Q4 or a smaller one at Q8?

Usually the bigger model at Q4. More parameters give the model more capability, and Q4 keeps quality high enough that the larger model still wins for most everyday tasks.

Why does VRAM matter more than gaming performance?

Local models need to sit entirely in VRAM to run fast. A card with lots of memory can hold bigger models even if it posts fewer gaming frames than a faster card with less VRAM.