What Is LLM Inference? Why It Is Limited by Memory Bandwidth, Not Cores

Run a chatbot or a coding assistant on your own machine and you quickly learn that the number printed on the box, the core count, is not what decides how fast the words appear. LLM inference is bound by how quickly model weights travel from memory to the processor, which is why two machines with identical core counts can feel worlds apart.

Quick Answer

LLM inference is the process of generating output from an already-trained model. Token generation is memory-bandwidth-bound: the speed limit is how fast weights move from memory each step, not raw core count. Adding cores without raising bandwidth delivers diminishing returns.

What inference actually does each step

Training builds the model. Inference is everything after that: you feed in a prompt and the model produces tokens, one at a time. To generate a single token, the system reads through the model's weights, the billions of numbers learned during training. Then it does it all over again for the next token.

That repeated full read is the catch. For every token, a large slice of the weights has to be pulled out of memory and into the compute units. The arithmetic itself is fast. The waiting is the bottleneck.

Why bandwidth beats cores

Memory bandwidth is how many gigabytes per second the processor can pull from memory. If your weights are 16GB and your effective bandwidth is 800GB/s, the hardware can physically read those weights only so many times per second, and that ceiling sets your tokens-per-second.

Cores help with parallel maths, but during single-stream token generation they spend much of their time idle, waiting for the next batch of weights to arrive. You can add cores all day; if the memory pipe stays the same width, the queue does not move faster. This is why high-bandwidth GPU memory and unified-memory designs punch so far above their core counts for this specific job. Machines built for this work, including the current crop of AI-focused desktops in the AI PC range, pair fast memory with capable accelerators for exactly that reason.

What this means when you are buying

If you want to run models locally, prioritise memory bandwidth and the amount of fast memory available, not the headline core figure. A GPU with wide, fast VRAM will generally generate tokens quicker than a CPU with many cores and ordinary system RAM. The amount of memory also caps which models fit at all: a model that does not fit in fast memory either spills to slower storage or simply will not load.

Quantisation, which stores weights at lower precision, helps on both fronts. Smaller weights mean less data to move per token and a smaller memory footprint, which is why quantised models often run noticeably faster on the same hardware. For broader build inspiration, the most popular desktop builds show how others are pairing memory and GPU horsepower.

Frequently Asked Questions

Does more RAM make local LLMs faster?

More RAM lets larger models fit, but capacity and bandwidth are different things. Speed comes from how fast that memory can be read, so a faster, narrower pool can outpace a large but slow one for token generation.

Why is my GPU faster than my CPU for this?

GPUs typically have far higher memory bandwidth than CPUs paired with standard system RAM. Since inference is bandwidth-bound, that wider, faster memory pipe translates almost directly into more tokens per second.

What is the single number to look for?

Memory bandwidth in gigabytes per second, alongside the amount of fast memory. Together they tell you how quickly weights can be read and whether a given model fits at all, which matters more than core count for inference.

Does quantisation reduce quality?

Light quantisation usually has a small effect on output quality while cutting memory use and boosting speed. Aggressive quantisation can degrade results, so most local users settle on a middle ground that fits their hardware comfortably.