Dropped From 50 to 2 Tokens a Second? Your Model Overflowed Into RAM

When a model spills from VRAM into system RAM, throughput collapses from 50+ tok/s to just 2-3 tok/s due to slow RAM bandwidth. Shrinking the model with Q4_K_M quantisation until it fits in VRAM is the only way to recover full speed.

Quick Bytes · 26 Aug 2026 · 5 min read · NexaForge ·

Your local model was humming along at 50 tokens a second, then a slightly longer prompt or a bigger context window dropped it to a crawl of 2 or 3. Nothing crashed, nothing errored, it just got 20 times slower. That collapse is the signature of a model that has overflowed into RAM: part of it no longer fits in your GPU's VRAM, so it spilled into system memory, and system memory is far too slow to feed a GPU at full speed.

Quick Answer

When a model exceeds your VRAM, the overflow runs in system RAM, which has a fraction of the bandwidth, so throughput craters from 50-plus tok/s to single digits. The only real fix is to make the model fit entirely in VRAM: use a more aggressive quantisation such as Q4_K_M, shrink the context window, or move to a smaller model.

Why a Few Hundred Megabytes Costs You 90 Percent of Your Speed

A GPU generates tokens fast because its VRAM bandwidth is enormous, often well over 500 GB/s on a modern card. System RAM, even fast DDR5, sits around 50 to 90 GB/s and has to talk to the GPU over the PCIe bus on top of that. When the weights and the key-value cache all fit in VRAM, the GPU never waits. The moment even one layer spills to RAM, every token has to wait on that slow path, and because token generation is sequential, the whole pipeline runs at the speed of the slowest memory in the chain.

This is why the drop is so brutal and so sudden. It is not a gentle slowdown proportional to how much overflowed. A model that is 95 percent in VRAM and 5 percent in RAM can still run at near-RAM speed, because the GPU keeps stalling on that 5 percent. Tests on consumer hardware regularly show models running roughly 30 times slower once they spill out of VRAM.

How to Get Your Speed Back

Quantise harder

Quantisation shrinks the model by storing weights at lower precision. Q4_K_M, the default in Ollama and llama.cpp, is the sweet spot most people reach for: roughly four bits per weight with very little quality loss for most work. A 7B model lands near 6 GB at Q4_K_M and a 70B near 42 GB. If your current model spills, dropping from a heavier quant to Q4_K_M is usually the first and cheapest move.

Cut the context window

The key-value cache grows with context length, and at long contexts it can quietly eat gigabytes. If you are loading a 16K or 32K context you rarely use, drop it. Enabling Flash Attention in current builds of Ollama or LM Studio also trims context VRAM overhead noticeably at long context lengths.

Step down a model size

If a model only just overflows at the quant and context you need, the honest answer is the next size down. A 7B at Q4_K_M that lives entirely in VRAM will out-generate a 13B that spills, every time, by an order of magnitude.

Add VRAM

If you keep hitting the ceiling, the hardware is the limiter. More VRAM means bigger models, longer contexts, and headroom to stop quantising so hard. The right card depends on the models you run, and the AI PC range at Evetech is stocked with machines configured for local-inference demands.

Spotting the Overflow Before It Bites

Watch your VRAM usage as the model loads. If the loader reports part of the model offloaded to CPU or system RAM, you have found your problem. Most local runners will tell you how many layers went to the GPU versus the CPU. The goal is every layer on the GPU. When your build can keep the whole model resident, the speed is consistent and predictable, which is the whole point of running locally. If you are pricing a machine for serious local AI work, the PC best sellers at Evetech show which GPU and memory pairings SA buyers are actually picking.

Frequently Asked Questions

How do I know if my model overflowed into RAM?

Check your loader's output as the model loads. If it reports layers offloaded to CPU or system RAM, or your GPU VRAM is maxed while system RAM use climbs, the model has spilled. The tell-tale symptom is a sudden collapse in tokens per second.

Will Q4_K_M hurt quality much?

For most tasks the quality loss from Q4_K_M is minor and hard to notice in normal use. It is the standard balance point between size and quality, which is why Ollama and llama.cpp default to it.

Is more RAM the answer?

No. Adding system RAM lets a bigger model load, but it runs at slow RAM speed, which is the exact problem. To recover speed you need the model in VRAM, so more VRAM is what matters.

Does context length really use that much memory?

Yes, at longer contexts. The key-value cache scales with context length and can consume several gigabytes. Trimming an oversized context window or enabling Flash Attention frees real VRAM.

If your model keeps spilling out of VRAM, the hardware is asking for more headroom. Compare GPU and memory configurations built for local inference on the AI PC range at Evetech and stop fighting the slow path.

For most tasks the quality loss from Q4_K_M is minor and hard to notice in normal use. It is the standard balance point between size and quality, which is why Ollama and llama.cpp default to it.

No. Adding system RAM lets a bigger model load, but it runs at slow RAM speed, which is the exact problem. To recover speed you need the model in VRAM, so more VRAM is what matters.

Yes, at longer contexts. The key-value cache scales with context length and can consume several gigabytes. Trimming an oversized context window or enabling Flash Attention frees real VRAM.