70B Model Crawling at a Few Tokens a Second on CPU-Only? Here Is Why

A 70B model running on CPU only crawls at one to three tokens a second for a simple reason: the processor cannot read the model's weights from system RAM fast enough. Every token forces the machine to stream tens of gigabytes of parameters, and ordinary DDR memory bandwidth is a fraction of what a graphics card delivers. The fix is not a faster CPU. It is getting at least some of those weights onto a GPU.

Quick Answer

CPU-only inference on a 70B model is slow because it is memory-bandwidth bound, not compute bound. Desktop DDR5 delivers tens of GB/s while a modern GPU pushes well over 1,000 GB/s, so offloading even part of the model to a GPU lifts throughput from a painful 1 to 3 tok/s into the usable 8 to 18 tok/s range.

Why bandwidth, not clock speed, is the wall

To generate one token, the model has to read every weight it needs for that step. A 70B model quantised to 4-bit still occupies roughly 40GB. At that size, the bottleneck is how quickly the hardware can move 40GB out of memory and into the maths units, repeated for each token.

A typical desktop pushes somewhere in the region of 50 to 90 GB/s of system memory bandwidth. A current GPU pushes far more: an RTX 5090 sits around 1,792 GB/s, and the older RTX 4090 around 1,008 GB/s. Because token generation scales almost directly with memory bandwidth, that gap is exactly why the same 70B model that crawls on a CPU runs an order of magnitude faster once the weights live in GPU VRAM. The AI PC range at Evetech lists machines configured around the GPU bandwidth that local inference demands.

What partial GPU offload actually does

You do not need enough VRAM to hold the entire model. Runtimes like llama.cpp and Ollama let you offload a set number of transformer layers to the GPU while the rest stay in system RAM on the CPU. The layers on the GPU run at GPU bandwidth, dragging the average speed up sharply.

Real numbers make the point. Llama 2 70B at a 4-bit quant on an RTX 4090 reaches roughly 18 tok/s with 40 layers offloaded into about 23GB of VRAM. Drop to 20 layers and only 12GB of VRAM, and it falls to around 8 tok/s, still far better than CPU alone. Put the whole thing on a high-bandwidth card and a 70B Q4 model can hit 40 to 50 tok/s. The practical takeaway is that the more layers you can fit on the GPU, the faster every token comes back.

Making local 70B usable in practice

If you are running models for iterative coding or long chats, three tok/s makes every reply a chore, while ten or more feels responsive. Prioritise VRAM capacity so you can offload as many layers as possible, then memory bandwidth so the layers you do offload run fast. More system RAM helps you load the model at all, but it will not rescue your speed on its own. For a build aimed squarely at this workload, the PC best sellers at Evetech shows which machines with real GPU headroom SA buyers are actually ordering.

Frequently Asked Questions

Why is my 70B model only doing 1 to 3 tokens a second on CPU?

Because token generation is limited by memory bandwidth, and system RAM moves data far slower than GPU VRAM. The CPU is starved waiting for 40GB of weights to stream through for every token.

Will a faster CPU fix it?

Only marginally. The bottleneck is memory bandwidth, not core count or clock speed, so a faster processor barely moves the needle. Adding a GPU and offloading layers is the real lever.

How much VRAM do I need for a 70B model?

To hold a 4-bit 70B model fully in VRAM you need roughly 40GB or more. With less, you offload only some layers to the GPU and keep the rest on the CPU, which still helps a great deal.

What token speed feels usable?

Around 8 to 10 tok/s reads at a comfortable pace, and 15 plus feels snappy for coding. Anything under 5 tok/s makes interactive sessions frustrating.

Does more system RAM speed it up?

It lets you load larger models, but it does not raise tokens per second on its own. Speed comes from bandwidth, so GPU offload is what moves the number.