How do I reduce LLM latency?

Generate fewer output tokens first: OpenAI's latency guide estimates that cutting output tokens by 50% cuts latency by about 50%, while cutting the prompt by 50% saves only 1 to 5%. Then stream responses, use the smallest model that passes your evals, lower reasoning effort, cache the stable part of the prompt, run independent calls in parallel, and for hard real-time needs use a fast provider such as Cerebras (about 3,000 tokens per second on gpt-oss-120b) or Groq (about 500).

What causes slow LLM responses?

Mostly output length. Models generate tokens one at a time, so a 1,000-token answer takes about ten times longer than a 100-token one at the same speed. Reasoning models add hidden thinking tokens before the visible answer. Other causes are large models, long uncached prompts, sequential chains of calls, and queueing on shared capacity at peak times.

Does prompt caching reduce latency?

Yes, for long repeated prompts. Anthropic documents that prompt caching reduces processing time as well as cost for prompts with consistent prefixes, with 5-minute and 1-hour cache lifetimes. Put stable content (system prompt, tools, documents) first and the changing user message last so the cached prefix is reused.

Are fast modes worth the price?

Only when speed is the product. Anthropic's fast mode for Claude Opus 5.5 costs $8/$40 per million tokens against the standard $4/$20, for up to 2.5x faster output. OpenAI's Fast mode costs 2x and Ultrafast 6x the standard rate. For most apps, shorter outputs and a smaller model deliver a bigger speed-up at no extra cost.

Quick Answer

How to Reduce LLM Latency in 2026: 9 Fixes That Work

Published: October 13, 2026

The short answer

Latency in an LLM app is mostly the time spent generating output tokens, so the biggest wins come from generating fewer of them and generating them faster. OpenAI’s latency guide puts numbers on it: halve the output and you roughly halve the latency; halve the prompt and you save only 1–5%. In order of payoff, as of October 13, 2026:

#	Fix	Typical effect	Cost
1	Shorter outputs (length limits, terse formats)	~linear: −50% tokens ≈ −50% time	Free
2	Stream the response	First words in well under a second	Free
3	Lower reasoning effort	Removes hidden thinking tokens	Usually cheaper
4	Smaller or faster model	Several-fold faster output	Usually cheaper
5	Prompt caching	Faster processing of long repeated prefixes	Cheaper
6	Parallel calls	Total time = slowest call, not the sum	Free
7	Fewer round trips	Combine sequential steps into one call	Free
8	Fast inference provider	~500–3,000 tokens/s on open models	Varies
9	Paid speed tiers	Up to 2.5x (Anthropic) or 6x (OpenAI Ultrafast)	2x–6x price

Measure the right two numbers

Track time to first token (TTFT) and output tokens per second separately. TTFT is what users feel in chat; tokens per second decides how long a full answer or an agent step takes. A slow TTFT points at queueing, long uncached prompts or reasoning; slow throughput points at model size or provider.

The fixes in detail

1. Generate fewer tokens. Ask for concise answers, return compact JSON with short keys, and set max_tokens. OpenAI’s guide calls output generation “almost always the highest latency step.”

2. Stream. Streaming does not shorten generation, but users start reading at the first token. It is the cheapest perceived-latency fix there is.

3. Lower reasoning effort. Reasoning models think before they answer. Claude Opus 5.5 defaults to medium effort; GPT-6 Sol runs from none to max. For classification, extraction and short replies, the lowest effort level that passes your evals removes most of the delay.

4. Use a smaller model. OpenAI notes that smaller models are usually faster and cheaper and can match larger ones with better prompts, few-shot examples or fine-tuning. Route easy requests to models such as Claude Haiku 5.5 or GPT-6 Luna (both $0.10/$0.50 per million tokens) and reserve frontier models for hard ones.

5. Cache the prompt prefix. Anthropic’s prompt caching lets requests resume from a cached prefix, which “significantly reduces processing time and costs.” Put tools, system prompt and reference documents first and the user’s message last. Our prompt caching explainer covers the pricing.

6. Parallelize. Independent steps — retrieving from three sources, checking three policies — should run at once.

7. Make fewer requests. A chain of five sequential calls pays five round trips and five TTFTs. Where quality allows, ask for several outputs in one structured response.

8. Use a fast provider for open models. Groq serves gpt-oss-120b at about 500 tokens per second for $0.15 input, $0.075 cached and $0.60 output per million tokens, with a 131,072-token context. Cerebras serves the same model at about 3,000 tokens per second for $0.35/$0.75. Compare them in best providers for real-time inference.

9. Pay for speed only where it matters. Anthropic’s fast mode for Claude Opus 5.5 is $8/$40 per million tokens (standard $4/$20) for up to 2.5x speed. OpenAI’s Fast mode costs 2x the standard rate, and Ultrafast 6x — GPT-6 Astra Ultrafast is $60/$300 for up to 300 tokens per second. Full rates are on our current API prices page.

Two more things to check

Region. Calling an endpoint on another continent adds a network round trip to every request; pick a region near users, and see EU-hosted options if you also need data residency.
Do you need an LLM at all? OpenAI’s last principle is “don’t default to an LLM”: hard-coded responses, cached answers to frequent questions and classic search are faster than any model.

Last verified: October 13, 2026.

The short answer

Measure the right two numbers

The fixes in detail

Two more things to check

Sources