Local AI Running Too Slow? How VRAM and RAM Affect Speed

When local AI runs too slow, the cause is nearly always the same: the model is bigger than the VRAM on your graphics card, so the software shoves the overflow into ordinary system RAM. That single decision wrecks speed, because the memory bandwidth between the GPU and its own VRAM is many times higher than the bandwidth to system RAM. The model still runs, but tokens crawl out instead of streaming. Fix the size mismatch and the speed comes back.

Quick Answer

Slow local AI is almost always VRAM overflow. If the model does not fit entirely in your GPU's VRAM, the runner offloads layers into system RAM, where the far lower bandwidth slashes your tokens per second. The fix is to match model size to your VRAM: use a smaller model or a tighter quantisation so everything fits on the card.

Why Overflow Kills Speed

A language model is a stack of layers, and each layer needs to be read from memory for every token it generates. When all the layers sit in GPU VRAM, the card reads them at high bandwidth and produces tokens fast. The moment the model is too large, the runner keeps some layers in VRAM and pushes the rest into system RAM connected to the CPU.

Now every token has to wait on the slow layers. System RAM bandwidth is a fraction of GPU memory bandwidth, so those offloaded layers become the bottleneck for the entire model. It does not matter that most of the model is on the fast card; the slowest part sets the pace. This is why a model that is only slightly too big can feel dramatically slower than one that fits cleanly.

How To Make It Fit

There are two practical levers, and you can pull either or both. The first is to run a smaller model. A 7B or 8B model fits comfortably on an 8GB card, while a 13B or 14B model usually wants 12GB or more. Matching the model's parameter count to your VRAM is the cleanest fix and often the only one you need.

The second lever is quantisation, which compresses the model's weights to use less memory at a small cost to quality. A model at Q4 takes far less VRAM than the same model in full precision, often enough to bring a model that was overflowing back fully on-card. If you are running a 13B model at high precision and it spills, dropping to a Q4 build frequently lets the whole thing fit and run fast. You can see the kind of VRAM the cards in the AI PC range at Evetech carry, which tells you how large a model you can host before overflow begins.

Check Before You Blame The Hardware

Before assuming you need a bigger card, confirm overflow is actually the problem. Watch your GPU's VRAM usage while the model loads. If it pins to the maximum and your speed is poor, you are overflowing and a smaller or more quantised model will help immediately. If VRAM is not full and it is still slow, look at whether the model is even running on the GPU at all, since a misconfigured runner can default to the CPU.

When The Answer Is More VRAM

Sometimes you genuinely need a model larger than your card can hold, and no amount of quantisation gets you both the size and the quality you want. That is the point where a GPU with more VRAM is the real fix, since it lets the bigger model sit entirely on the card and run at full speed. For a sense of what complete machines built around higher-VRAM cards look like and cost, the PC best sellers give a quick reference on balanced local-AI configurations.

Frequently Asked Questions

How do I know if my model is overflowing VRAM?

Watch your GPU's VRAM usage as the model loads. If it fills to the maximum and generation is slow, the runner is offloading layers into system RAM. A model that fits with headroom and still runs fast confirms you are within your card's limit.

Will a smaller model really run faster?

Yes, if the smaller one fits in VRAM while the larger one overflows. Speed is set by the slowest layers, so a model that lives entirely on the card runs far faster than a bigger one forced to use slow system RAM.

Does quantisation hurt quality?

A little, but often less than you expect. Dropping to Q4 trims memory use sharply for a modest quality cost, and for most everyday local AI tasks the trade is worth it because it keeps the whole model on the fast GPU memory.

Is more system RAM the answer to slow local AI?

Not usually. System RAM lets a big model load, but it runs slowly because of low bandwidth. The real speed comes from fitting the model in GPU VRAM, so a card with more VRAM, or a smaller model, is the better fix.