Prompt caching and cutting LLM latency

Two things make an LLM feel slow: the wait before anything happens, and the wait for it to finish. Almost every latency win is really an attack on one of those two, and the biggest one is free — prompt caching cuts both the cost and the startup delay at the same time. This guide walks the levers in the order I actually reach for them, from the free structural wins to the tradeoffs you have to measure. It's the latency companion to cutting your LLM costs, and the two overlap constantly, because the same caching that shrinks your bill also shrinks your time-to-first-token.

Split latency into the two numbers that matter

Before optimizing anything, separate the wait into time-to-first-token (TTFT) — how long until the first output appears — and total latency — how long until the whole response is done. These are different problems with different fixes, and conflating them sends you optimizing the wrong one.

The reason the split matters: which one the user feels depends entirely on how the output is consumed. If a person reads the response as it arrives — a chat reply, an assistant, generated copy on a screen — TTFT dominates the felt experience, because once words start flowing the human is occupied reading. If the output is consumed by a machine all at once — a block of JSON your code parses before it can act — TTFT is irrelevant and total latency is the whole game, because nothing can happen until the last token lands. Decide which your feature is before you spend an hour tuning the wrong number.

Prompt caching: faster and cheaper at once

Processing the input prompt is real work, and it happens up front, before the model emits its first token. So a long prompt directly inflates TTFT. If a big stable block of context repeats across requests — a system prompt, a reference document, few-shot examples, a tool list — prompt caching lets the provider reuse the work it already did on that prefix instead of reprocessing every token. You get a lower bill and a faster first token from the same change.

The one rule that makes or breaks it: caching is a prefix match. The provider caches from the start of the prompt up to a marked point, and any change before that point invalidates everything after it. So structure the prompt with stable content first and volatile content last:

[ tools ]            ← stable, goes first
[ system prompt ]    ← stable
[ retrieved docs ]   ← stable across the session
  --- cache marker ---
[ the user's turn ]  ← changes every request, goes last

The classic mistake is interpolating something volatile — a timestamp, a request ID, the current date — into the top of the system prompt. That single changing string means nothing after it ever caches: no error, just a TTFT that never improves and a bill that never drops. Providers differ in the details — how long a cache entry lives, minimum prefix sizes, whether you mark cache points explicitly or the provider infers them — so read your provider's specifics, but the prefix-match principle is universal. Verify it's working by reading the cached-token count off the response; if it stays at zero across requests that should share a prefix, something volatile is leaking into the front.

Stream for perceived latency

For anything a human reads, streaming is the highest-leverage move you have. Instead of waiting for the entire response, you send each token as it's generated, so the user sees words appearing within a few hundred milliseconds instead of watching a spinner for the whole duration. The total generation time is exactly the same — you're improving perceived latency, not real latency — but perception is what a user actually experiences. A three-second response that starts rendering at 400ms feels responsive; the same three seconds behind a blank spinner feels broken.

Streaming only helps when something can consume a partial response. A reader can. A JSON parser that needs the whole object generally cannot, so for machine-consumed structured output streaming buys you little and total latency is what you optimize. If you're generating structured data for a UI, one useful middle path is to stream a human-readable part first and produce the strict machine part separately — see structured output and tool use for how to keep the strict parts strict.

Set a latency budget and size the model to it

Decide up front what "fast enough" means for this feature — a target TTFT and a target total — and treat it as a budget you spend across the pipeline. Every stage costs time: retrieval, the prompt you build, the model you pick, the tokens you ask it to produce. A budget turns vague "it feels slow" into "retrieval is eating 600ms of our 1.5s target," which is something you can fix.

Model size is the biggest single line in that budget. Smaller models generate tokens faster and start sooner, so on well-scoped work they're quicker and cheaper. The tradeoff is capability, so match the model to task difficulty and validate the swap on real data rather than a vibe — the full decision is in big model vs small model. Two structural moves that spend the budget well: cap output length, because output tokens are generated one at a time and a response half as long finishes in roughly half the time, so ask for exactly what you need and no more; and fan out cheap, escalate rarely, running the fast model on the common case and sending only the hard minority to a bigger, slower tier — most requests get the fast path and only the few that need it pay for the smart one.

Parallelize what doesn't depend on itself

If your feature makes several model calls, the naive version runs them in sequence and pays the sum of their latencies. Any calls that don't depend on each other's output should run concurrently instead, so you pay the slowest one rather than the total. Classifying an item, retrieving related context, and drafting a summary that doesn't need the classification can all fire at once. This is the single biggest win in multi-step and agentic flows, where sequential chaining quietly stacks four round-trips into one long wait — see the building with LLMs field guide for how these pipelines are structured.

A speculative variant helps when a later step usually takes a predictable path: kick off the likely next call before you've confirmed you need it, and throw the result away if you were wrong. You trade some wasted tokens for a shorter wall-clock wait. It's worth it when latency matters more than the marginal cost and your guess is right most of the time — and not worth the complexity when it isn't. Measure before assuming it helps.

Measure p50 and p95, never the average

You cannot improve latency you don't measure, and the average is the wrong measurement. Record two numbers on every call — TTFT and total time — and report them as percentiles. p50 (the median) is your typical experience; p95 is what a meaningful slice of users actually hits, and it's usually far worse than the mean, because token counts, cache hits, and provider load all vary call to call. The average smears the slow tail into a comfortable-looking number while real users sit in the p95 that drives the complaints.

Log input, output, and cached token counts next to the timings, because that's how you diagnose a slow call instead of just noticing it: a bad TTFT with zero cached tokens is a caching miss; a bad total with a huge output is a length problem; both fine except during one window is the provider having a moment. Watch p95 as you ship changes — a fix that improves the median while the tail gets worse is often a net loss for how the app actually feels. The discipline is the same one that governs cost: instrument first, change one thing, watch the number move, keep what worked. Fast and cheap are usually the same optimization viewed from two sides, and the prompt structure that earns you cache hits is where both start — which is also where good production prompt engineering begins.

FAQ

How does prompt caching make an LLM app faster?

Processing the input prompt takes real compute, and that work usually happens before the model emits its first token. When a big stable block of context repeats across requests — a system prompt, a document, a tool list — prompt caching lets the provider reuse the work it already did on that prefix instead of reprocessing every token. That cuts both cost and the time-to-first-token, because the model skips straight to the part that changed. The catch is that caching is a prefix match: put stable content first, volatile content last, or nothing caches.

What's the difference between TTFT and total latency, and which matters?

Time-to-first-token (TTFT) is how long the user waits before anything appears. Total latency is how long until the full response is done. They matter for different reasons. For anything a person reads as it arrives — chat, assistants, generated copy — TTFT is what the user feels, and streaming tokens as they generate makes a slow total response feel fast. For machine-consumed output that another step parses all at once — JSON you have to fully receive before acting — total latency is what counts and streaming buys you nothing. Budget the one your use case actually cares about.

Does streaming actually make responses faster?

It makes them feel faster without changing the total time at all. Streaming sends each token as it's generated instead of waiting for the whole response, so the user sees words appearing within a few hundred milliseconds instead of staring at a spinner for the full duration. The generation takes exactly as long either way — you're improving perceived latency, not real latency. It's the single highest-leverage move for any interface a human reads, and it does nothing for output a machine consumes in one shot.

Should I use a smaller model to reduce latency?

Often yes — smaller models generate tokens faster and start responding sooner, so on well-scoped work like classification, extraction, and routing they're both cheaper and quicker. The tradeoff is capability, so match the model to how hard the task actually is and validate the swap on your own data. A strong pattern is to run the fast model on the common case and escalate only the hard minority to a bigger one, so most requests get the fast path and only the few that need it pay the slower, smarter tier.

How do I measure LLM latency properly?

Record two numbers on every call — time-to-first-token and total time — and report them as percentiles, not averages. p50 is your typical experience; p95 is what a meaningful slice of users actually hits, and it's usually far worse than the mean because token counts and provider load vary. Averages hide the slow tail that drives complaints. Log input, output, and cached token counts alongside the timings so you can tell whether a slow call was a caching miss, a long output, or the provider having a bad moment.