Which LLM marketplace has the lowest latency for streaming responses?

None is fastest by itself: a marketplace is only as fast as the provider it routes to, so the winner is the one that lets you route by latency. As of November 4, 2026, OpenRouter is the most configurable — provider.sort set to latency, a preferred_max_latency threshold at p50, p75, p90 or p99, and the :nitro suffix that sorts by throughput and admits priority-tier endpoints. Vercel AI Gateway sorts by time to first token ('ttft') or throughput ('tps'); Hugging Face Inference Providers defaults to the fastest provider by throughput (:fastest).

Do LLM marketplaces add a markup to provider prices?

Not on tokens. OpenRouter passes through provider pricing with no inference markup but charges a fee when you buy credits, and bring-your-own-key usage above $25,000 a month (pay-as-you-go) costs 5% of the normal price. Vercel AI Gateway charges no markup and no platform fee on tokens, including with your own keys. Hugging Face says Inference Providers add no extra markup on provider rates. All as stated in each vendor's docs on November 4, 2026.

How do I route an OpenRouter request to the lowest-latency provider?

Add a provider object to the request: {"sort": "latency"} tries providers in order of lowest latency, bypassing default routing. To keep OpenRouter's balanced default but prefer fast providers, set preferred_max_latency instead, for example {"p90": 2} for 2 seconds at the 90th percentile. Appending :nitro to the model slug sorts by throughput instead.

Is a marketplace slower than calling the provider directly?

It adds a network hop, so a direct call to the same provider in the same region is never slower. The marketplace wins when it can fail over away from a slow or rate-limited provider: OpenRouter lowers a provider's traffic share after a burst of 429s or a latency jump, and Vercel AI Gateway adjusts provider order for recent health. Measure time to first token both ways from your own servers before choosing.

Quick Answer

Lowest Latency LLM Marketplace for Streaming Responses

Published: November 4, 2026

The short answer

A marketplace cannot be faster than the provider behind it, so pick the one that routes on latency the way you need. As of November 4, 2026:

OpenRouter — best control: sort by latency, set a p50–p99 latency threshold, or use :nitro for the highest-throughput endpoint.
Vercel AI Gateway — best if you build on the AI SDK: sort by time to first token (ttft) or throughput, zero token markup.
Hugging Face Inference Providers — best for open-weight models: routes to the fastest provider by default (:fastest), including Cerebras and Groq.

If you need one specific fast provider, go direct — see our low-latency inference providers hub.

The comparison

	OpenRouter	Vercel AI Gateway	Hugging Face Inference Providers
Default routing	Balances effective price, throughput, TTFT, 429 rate, reliability, tool-call quality	Default provider priority adjusted for recent uptime and latency	Fastest provider by throughput (`:fastest`)
Latency control	`sort: "latency"`; `preferred_max_latency` (p50/p75/p90/p99)	`sort: "ttft"`	Not a separate latency sort; `:fastest` is throughput
Throughput control	`sort: "throughput"`, `:nitro` (adds priority-tier endpoints)	`sort: "tps"`	`:fastest` (default)
Price control	`sort: "price"`, `:floor` (adds flex tier), `max_price`	`sort: "cost"`	`:cheapest` (lowest output price)
Pin or exclude providers	`order`, `only`, `ignore`, `allow_fallbacks`	`order` (keeps failover), `only` (hard filter)	`:preferred` (your settings order)
Token markup	None; fee on credit purchases; BYOK 5% above $25K/mo	None, including BYOK	None
Best for	Any model, fine-grained routing	AI SDK / Vercel apps	Open-weight models, HF accounts

Behaviour from each vendor’s routing docs, read November 4, 2026.

OpenRouter: the most latency knobs

Default routing already weighs time to first token, throughput and reliability from recent production traffic, and reacts within minutes. Providers whose 429s or errors are far worse than their peers get no traffic until they recover, and follow-up turns stay on the same provider while its prompt cache is warm.

For a latency-critical stream:

{"model": "your-model", "stream": true,
 "provider": {"preferred_max_latency": {"p90": 2}}}

preferred_max_latency keeps default routing but prefers providers that meet the threshold — it does not guarantee it. sort: "latency" replaces default routing with a strict latency order. Use p90 or p99 for user-facing features, because p50 hides the slow tail.

Watch: :nitro sorts by throughput (tokens per second), not time to first token. For chat UIs, TTFT matters more; for long generations, throughput does.

Vercel AI Gateway: TTFT sorting with zero markup

Set providerOptions.gateway.sort to 'ttft' to try the provider with the lowest time to first token first, or 'tps' for throughput. order prefers providers while keeping failover; only is a hard filter that fails instead of falling back. Compliance filters — zero data retention, disallow prompt training, regional inference — keep every qualifying provider eligible, which is better for latency than pinning one. Tokens are billed at provider list price with no markup or platform fee; budgets can cap spend per team, project, key or member.

Hugging Face Inference Providers: fastest by default

One Hugging Face token reaches Cerebras, Groq, Together, Fireworks, Baseten, DeepInfra, Nscale, Scaleway and others through an OpenAI-compatible API. The default policy is the fastest available provider by throughput; append :cheapest or :preferred to the model id to change it (for example openai/gpt-oss-120b:fastest). There is no extra markup on provider rates, and PRO users and Team/Enterprise organizations get monthly included credits.

Picks by situation

Voice agent or chat UI, every 100 ms counts: OpenRouter with preferred_max_latency at p90, or Vercel with sort: 'ttft' — then test the direct provider too.
Long code generations: throughput sorting (:nitro, sort: 'tps', HF :fastest).
Open-weight model on Cerebras or Groq without separate accounts: Hugging Face Inference Providers.
Many models, one bill, failover: OpenRouter; compare gateways in best LLM gateway.

How to test before you commit

From your production region, stream 200 identical prompts through each marketplace and directly to the fastest provider.
Record TTFT p50/p90 and tokens per second separately.
Repeat at peak hours — rate limits, not raw speed, cause most slow responses.
Check which provider served each request (all three return provider metadata) so you know what you measured.

Last verified: November 4, 2026.