Low Latency Inference Providers 2026: Groq, Cerebras & More
The short answer
For the lowest latency on open models, use Cerebras; for low latency at the lowest price, use Groq; for large frontier open models, use Fireworks’ Fast variants. Latency in an LLM app is mostly output generation time, so the number that matters is output tokens per second on the model you actually need. As of October 14, 2026:
| Provider | Fast models (shared tier) | Output speed | Price per 1M tokens (in / out) | Pick it when |
|---|---|---|---|---|
| Cerebras | gpt-oss-120b · Qwen 3.8 27B | ~3,000 tok/s · ~1,850 tok/s | gpt-oss-120b $0.35 / $0.75 | Speed is the product (voice, live UI, agent loops) |
| Groq | gpt-oss-120b · gpt-oss-20b | ~500 tok/s · ~1,000 tok/s | $0.15 / $0.60 · $0.075 / $0.30 | Fast and cheap on gpt-oss |
| Fireworks AI (Fast) | Kimi K3 Fast · GLM 5.3 Fast | targets 100+ tok/s | $4.50 / $22.50 · $2.10 / $6.60 | You need a frontier-size open model to be quick |
| SambaNova Cloud | gpt-oss-120b · MiniMax-M3 · DeepSeek-V3.1 | not published per model | $0.22 / $0.59 · $0.60 / $2.40 · $3 / $4.50 | A second fast source for gpt-oss or MiniMax |
| Closed models | Claude Haiku 5.5 · GPT-6 Luna | ~237 tok/s · ~135 tok/s (Artificial Analysis) | $0.10 / $0.50 each | You need a closed model and low latency |
Cerebras and Groq speeds are each vendor’s own published figures; closed-model speeds are Artificial Analysis medians at max effort. This page is the low-latency spoke of our best LLM inference providers ranking.
Picks by use case
Voice agents and real-time UI: Cerebras. At ~3,000 tokens per second, a 300-token reply on gpt-oss-120b streams out in about a tenth of a second. The catch is catalog size: the shared tier serves two models (gpt-oss-120b and Qwen 3.8 27B); everything else is dedicated inference. Context is 65K tokens on the free tier and 131K on paid.
Cost-sensitive real-time apps: Groq. Groq charges $0.15/$0.60 for gpt-oss-120b — less than half Cerebras’ price — at about 500 tokens per second, with a 131,072-token context and 65,536-token max completion. gpt-oss-20b runs at about 1,000 tokens per second for $0.075/$0.30. Developer-plan limits are 250K tokens per minute and 1K requests per minute for both.
Agents on frontier open models: Fireworks Fast. Fast is a routing mode, not a different model: Fireworks says quality is the same and Fast variants aim for 100+ tokens per second. You pay 50% above standard (Kimi K3 $3/$15 → Fast $4.50/$22.50; GLM 5.3 $1.40/$4.40 → Fast $2.10/$6.60). Separately, Fireworks’ Priority tier (Kimi K3 $3.75/$18.75) does not speed up output; it reduces the chance of 503 load-shedding at peak.
A closed model with low latency: Claude Haiku 5.5. Artificial Analysis measures Haiku 5.5 at about 237 output tokens per second against 135 for GPT-6 Luna, both at $0.10/$0.50 per million tokens. Watch effort settings: at max effort Haiku 5.5 reasons for a long time before the first visible token, so use a low or medium effort for interactive work. See Claude Haiku 5.5 vs GPT-6 Luna vs Gemini 3.8 Flash.
Paid speed tiers on frontier closed models: last resort. Anthropic’s fast mode for Claude Opus 5.5 costs $8/$40 per million tokens (2x) and runs on the Claude API only; OpenAI’s Ultrafast for GPT-6.1 Sol costs $12/$60 (6x). See current API prices.
How to test latency before you commit
- Measure time to first token and tokens per second separately, on your own prompt lengths. Long prompts raise time to first token; model size and provider decide throughput.
- Test at your peak hour. Shared tiers queue under load; a fast median can hide a slow p95.
- Turn reasoning down. Hidden thinking tokens are latency the user cannot see; a fast provider does not fix a model that thinks for 2,000 tokens.
- Check rate limits per organization. A fast endpoint that returns 429s at your traffic is not fast.
For the app-side fixes (shorter outputs, streaming, caching, parallel calls), see how to reduce LLM latency. For the broader price comparison, see leading serverless inference providers.
Last verified: October 14, 2026.