How do I self-host an OpenAI-compatible LLM on cloud GPUs with autoscaling?

As of November 9, 2026: run an open model on vLLM or SGLang, which both expose an OpenAI-compatible API, and put an autoscaler in front. On Kubernetes use llm-d (v1.0.0, October 20, 2026; SLO-aware autoscaling and prefix-cache-aware routing), KServe (v0.21.0, request-based autoscaling) or the vLLM production-stack Helm chart. Without Kubernetes, deploy the same vLLM container to a serverless GPU platform such as Modal, RunPod Serverless or Baseten, which scale replicas up and down for you.

Which is easier: llm-d, KServe or vLLM production-stack?

vLLM production-stack is the quickest start: one Helm chart with vLLM engines, a request router that keeps sessions on the replica holding their KV cache, LMCache offloading and Prometheus plus Grafana dashboards. KServe suits teams that already serve other ML models on Kubernetes and want one platform. llm-d is the most capable for large models and high traffic: prefill/decode disaggregation, wide expert parallelism, tiered KV cache and SLO-aware autoscaling.

Can a self-hosted LLM scale to zero?

Yes, but cold starts are the price. Serverless GPU platforms such as Modal bill per second and stop charging when no container runs; the first request after idle waits for a GPU and for the weights to load, which takes seconds to minutes for large models. On Kubernetes, keep at least one warm replica for user-facing traffic and scale to zero only for batch or internal endpoints.

What should I autoscale an LLM server on?

On queue depth, in-flight requests or KV-cache utilisation per replica, not CPU. LLM servers are GPU-bound and their CPU stays low while requests queue. llm-d and KServe both offer LLM-aware scaling signals; on a serverless platform set a concurrency target per replica so a new container starts before latency degrades.

Quick Answer

Self-Host an OpenAI-Compatible LLM With GPU Autoscaling

Published: November 9, 2026

The short answer

Self-hosting an OpenAI-compatible LLM with GPU autoscaling takes two layers: an inference engine that speaks the OpenAI API, and a control plane that adds and removes GPU replicas. As of November 9, 2026:

Engine: vLLM (v0.32.0, October 18, 2026) or SGLang (v0.5.23, November 2, 2026). Both serve /v1/chat/completions and /v1/completions, so existing OpenAI SDK code only changes its base_url.
Control plane on Kubernetes: llm-d for large models and heavy traffic, KServe if you already run ML on Kubernetes, vLLM production-stack for the fastest start.
No Kubernetes: a serverless GPU platform — Modal, RunPod Serverless or Baseten — runs your vLLM container and autoscales it.
Several clouds or regions: SkyPilot, with the caveat that its SkyServe serving library is still beta.

The comparison

Versions from GitHub and prices from vendor pages on November 9, 2026.

Option	What it is	Autoscaling	Licence / price	Best for
llm-d (v1.0.0, Oct 20, 2026)	Distributed serving stack on Kubernetes above vLLM/SGLang; CNCF sandbox, founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA	SLO-aware autoscaling, prefix-cache and load-aware routing	Apache-2.0	Big MoE models, high traffic, agent workloads
KServe (v0.21.0, Sep 25, 2026)	CNCF incubating model-serving platform with vLLM and llm-d backends	Request-based autoscaling	Apache-2.0	One platform for LLMs and classic ML
vLLM production-stack (Nov 7, 2026 chart)	Reference Helm chart: vLLM engines, router, LMCache, Prometheus + Grafana	Scale engines with Kubernetes autoscaling	Apache-2.0	Fastest path from one vLLM to a cluster
NVIDIA Dynamo (v1.6.2, Nov 5, 2026)	Datacenter-scale orchestration above vLLM, SGLang and TensorRT-LLM	Automatic scaling, disaggregated serving	Apache-2.0 (per README)	Multi-node NVIDIA fleets
Modal	Serverless containers with GPUs	Scales per request; per-second billing	Starter $0 + $30/mo credits; Team $250/mo + $100 credits	Python teams, spiky traffic
RunPod	GPU pods plus Serverless workers	Serverless workers scale on demand	H100 SXM pod $3.99/hr; Serverless billed by usage	Cheapest raw GPU-hours
Baseten	Managed custom-model deployments (Truss)	Autoscaling replicas	H100 $0.10833/min	Managed, production SLAs
SkyPilot (v0.14.2)	Launches replicas across clouds and regions	SkyServe load-balancing and autoscaling (beta)	Apache-2.0	Multi-cloud cost and capacity hunting

Step 1 — Get one replica right

Start with a single GPU node and vllm serve <model> (or SGLang’s server). Measure tokens per second and time to first token at the concurrency you expect. Fix tensor-parallel size, quantisation and context length here: autoscaling multiplies whatever one replica does, including its mistakes. Engine choice is covered in best LLM inference engines.

Step 2 — Pick the control plane your team can run

You have a platform team and Kubernetes: llm-d or KServe. llm-d’s well-lit-path guides ship benchmarked Helm recipes; its README cites up to 3x higher output throughput with prefix-cache-aware routing versus round-robin in a Red Hat deployment.
You want Kubernetes but a small surface: vLLM production-stack — engines, a router that sends a session back to the replica holding its KV cache, and dashboards in one chart.
You do not want Kubernetes: deploy the same container to Modal, RunPod Serverless or Baseten. You trade some per-hour cost for zero cluster operations. Vendor-by-vendor detail is in inference vendors with autoscaling and fast cold starts.

Step 3 — Scale on LLM signals

CPU utilisation is useless for LLM servers. Scale on queue length, in-flight requests per replica or KV-cache utilisation, and set the threshold below the point where your p95 latency breaks. On serverless platforms the equivalent knob is a target concurrency per container.

Step 4 — Decide what idle costs you

A replica of a large model can take minutes to start because the weights must be pulled and loaded. For user-facing traffic keep a minimum of one warm replica; cache weights on a fast volume close to the GPUs; scale to zero only for internal or batch endpoints. Modal’s pricing page frames the trade-off directly: you pay a higher per-hour rate but nothing for idle GPUs.

Step 5 — Put a gateway in front

Add API keys, per-team rate limits, request logging and a fallback route to a hosted API when your cluster is saturated. The OpenAI-compatible surface means one gateway can front both your self-hosted model and commercial providers — see best LLM gateways.

Step 6 — Check the economics

Self-hosting beats an API only at sustained utilisation. Run the numbers with the self-host vs API breakeven guide before committing to reserved GPUs; a per-token open-model host is often cheaper below a few hundred million tokens a day.

Last verified: November 9, 2026.