Inference Providers With Sub-Second Cold Starts (2026)
The short answer
Sub-second cold starts exist only where the platform restores a saved worker instead of booting a new one — or where the model never goes cold because a shared per-token API keeps it loaded. As of October 29, 2026:
| Option | Cold-start mechanism | Vendor’s claim | Price (list, H100 where shown) | Catch |
|---|---|---|---|---|
| Per-token APIs (Fireworks, Together AI, DeepInfra) | Model always warm, shared | No cold start | Per token | Catalog models only |
| Runpod Serverless | FlashBoot (retains worker state), cached models | ”Sub-200ms FlashBoot cold starts” | H100 flex worker $4.79/h, billed per second | Best on endpoints with steady traffic; a rarely called endpoint boots cold |
| Modal | CPU memory snapshots; GPU memory snapshots (alpha) | Initialisation-heavy functions start 3–10× faster | H100 $0.001097/s (≈ $3.95/h) | Snapshots only for deployed apps; GPU snapshots are alpha |
| Cerebrium | Container scaling plus memory and GPU snapshotting | Containers scale in 1–3 s | Per second by allocated GPU/CPU/RAM | Not sub-second by its own figure |
| Replicate | Fast-booting fine-tunes | Bills active time only for those | H100 $0.001525/s (≈ $5.49/h) | Other private models bill setup and idle time |
| Baseten | ”Fast cold starts” on Basic | No number published | Per minute per replica | Deploy and scale-up minutes are billed |
Prices and claims are from vendor pricing pages and documentation, read October 29, 2026.
This is a spoke of our broader comparison, best inference API vendors for autoscaling and fast cold starts, which covers billing, free tiers and deployment paths.
What “cold start” actually includes
A cold start is everything between a request arriving at an idle endpoint and the first token: scheduling a GPU, pulling the container image, loading weights into GPU memory, compiling or warming the engine, and then the first forward pass. Runpod’s documentation defines it the same way — starting the container, loading models and initialising the worker. Weight loading dominates for large models, which is why every sub-second claim relies on keeping state rather than reloading it.
Provider notes
Runpod Serverless. FlashBoot is on by default for new GPU and CPU endpoints. Runpod’s docs explain it “retains worker state after spin-down”, allowing a faster revival than a fresh boot, and say it is most effective where workers frequently cycle between active and idle. Pair it with cached models, which schedule workers onto machines that already hold your model files. Rates are per second, from $0.58 per hour for 16 GB cards to $10.65 for a B300.
Modal. Add enable_memory_snapshot=True and Modal saves the container’s memory after initialisation; later containers restore that state instead of re-importing libraries and re-running setup. Modal reports practical initialisation-heavy functions starting 3–10× faster. GPU memory snapshots, which also capture model weights already on the GPU, are an alpha feature enabled through experimental_options.
Cerebrium. Says containers scale up and down in 1–3 seconds and that memory and GPU snapshotting restore workloads faster still. Its distinguishing offer is guaranteed burst capacity without reserving GPUs around the clock.
Replicate. Public models are kept warm. Most private models run on dedicated hardware and bill setup and idle time, so they are a poor fit for spiky traffic unless they are fast-booting fine-tunes, which bill only active time.
How to choose
- Is the model in a per-token catalog? Use the API. Nothing is faster than a model that is never cold.
- Own weights, traffic arrives in bursts but the endpoint is called many times a day: Runpod Serverless with FlashBoot and cached models.
- Own weights, heavy Python initialisation: Modal with memory snapshots; test GPU snapshots before relying on them.
- Hard latency SLO on a large model: keep a minimum of one warm replica during business hours on any of these platforms and scale to zero at night. The warm replica costs less than the lost requests.
- Verify with your own test: leave the endpoint idle past its timeout, send one request, record time to first token, and repeat ten times.
For the cheapest per-token hosts see the cheapest inference provider for open-weight models.
Last verified: October 29, 2026.