Run a chatbot on your own PC and the single number that decides whether it feels usable is tokens per second. It measures how fast the model produces words once it starts answering, and the reason it is so often misunderstood is that the bottleneck is rarely your processor. Local AI generation leans on memory bandwidth far more than raw compute, which changes what you should actually spend money on.

Quick Answer

Tokens per second (tps) is how many chunks of text a model generates each second. Below 10 tps the output feels sluggish and you watch it type; at 30 tps or more it reads as near-instant. Because generation is memory-bandwidth-bound, faster memory raises this number more than adding compute cores.

What a token actually is

A token is a fragment of text, often a whole short word or a piece of a longer one. Roughly speaking, a token is about three-quarters of an English word, so 30 tokens per second works out near 22 words per second, comfortably faster than anyone reads. The model produces these one after another, and the rate at which they appear is your tps figure.

Why memory bandwidth, not cores, sets the ceiling

Generating each new token means the hardware has to read the model's weights out of memory. For a large model that is gigabytes of data shuffled for every single token. The arithmetic itself is quick; the time goes into moving those weights from memory to the chip. That makes generation memory-bandwidth-bound, which is a precise way of saying the limit is how fast data flows, not how many cores you have.

This is why a graphics card with fast GDDR memory often generates faster than a CPU with many cores but slower system RAM. It is also why two machines with similar core counts can post very different tps numbers: the one with quicker memory wins. When you size a build for local AI, prioritise memory speed and a GPU with enough fast VRAM to hold the model.

What counts as fast enough

For a back-and-forth chat, anything from 15 tps upward feels responsive, and 30 tps and above feels instant. For bulk jobs like summarising long documents, higher is simply quicker through the queue. The other half of the story is memory capacity: a model that overflows VRAM bleeds into slower system RAM and tps collapses sharply, so having enough fast memory to hold the entire model is as important as raw bandwidth.

If you are choosing hardware for local inference, the AI PC range at Evetech pairs capable GPUs with the memory headroom local models need, which makes it a practical reference point. For a broader view of what SA buyers are choosing across price bands, the current PC best sellers at Evetech shows where the value currently sits.

Frequently Asked Questions

What is a good tokens-per-second number for local AI?

For chat, 15 tps and up feels responsive and 30 tps or more reads as instant. Below 10 tps you will notice the model typing out its answer, which gets tiring in everyday use.

Does a faster CPU improve tokens per second?

Less than people expect. Generation is limited by how fast weights move out of memory, so memory bandwidth and a capable GPU raise tps far more than extra CPU cores.

Why does my model run slowly even on a strong PC?

The most common cause is the model not fitting in fast memory, so it spills into slower system RAM. Choosing a model that fits your VRAM, or a GPU with more VRAM, usually restores speed.

How do tokens relate to words?

A token is roughly three-quarters of an English word. So 30 tokens per second is around 22 words per second, well past normal reading speed.

Want local AI that answers without the wait? Explore the AI PC range at Evetech, built around fast memory and capable GPUs so your tokens-per-second stays high across real workloads.