Why Your Local Model Gets Slower the Longer You Chat

A model that felt instant at the start of a session crawls an hour later. Nothing was reinstalled and no setting changed. This is expected behaviour, and it has one cause.

Weights are allocated once when the model loads and never change size. The KV cache stores a key and value vector for every token processed so far, so it grows with the conversation. On a 14B model at Q4_K_M:

• ▸1,000 tokens — 0.22 GB of cache.
• ▸4,000 tokens — 0.90 GB.
• ▸8,000 tokens — 1.80 GB.
• ▸16,000 tokens — 3.59 GB.
• ▸128,000 tokens — 28.74 GB, which is more than twice the weights.

On a 12 GB card, that same model totals 10.67 GB at 8,000 tokens and 12.47 GB at 16,000. The first fits. The second does not, and the runtime responds by moving layers to system RAM — which is the slowdown you feel.

Because it is one. While everything fits, speed is constant. The moment the cache pushes a layer off the card, every token has to cross into system memory and back, and that side is roughly ten times slower. There is no gentle decline — there is fine, and then there is bad.

• ▸Start a new conversation. This releases the cache instantly and is the fastest fix by a wide margin.
• ▸Set a context length you can actually afford up front, rather than the largest the model supports.
• ▸Enable flash attention, which reduces peak attention memory and helps more as the context grows.
• ▸Quantise the KV cache. llama.cpp supports 8-bit and 4-bit cache, halving or quartering this term for a small quality cost on long contexts.
• ▸Summarise and restart. For long working sessions, carrying a summary into a fresh conversation is faster than dragging the whole history along.

Every figure here is computed the same way the calculators do it: weights are parameters times bits-per-weight divided by eight, plus a KV-cache term for your context, plus about a gigabyte of runtime overhead. Usable memory is roughly 90% of nameplate on a discrete card. Our cache estimate is architecture-agnostic and conservative for models using grouped-query attention, which most modern ones do — so treat these as an upper bound and trust a real run over them.