How do I deploy a fine-tuned LLM to production?

As of November 9, 2026: first decide whether to merge your LoRA into the base model (one fine-tune) or serve it as an adapter (many fine-tunes on one base). Then pick a host: Fireworks for managed LoRA with live merge or multi-LoRA ($8 per hour for an H100 or H200), Together AI dedicated endpoints, Baseten for any custom model billed per minute, or vLLM on your own GPUs. Put an OpenAI-compatible endpoint in front, gate the release on a frozen eval set, and roll out behind a canary.

Can I serve a fine-tuned LoRA serverless and pay per token?

Only in narrow cases. Together AI's documentation says serverless LoRA inference, including Serverless Multi-LoRA, has been discontinued and adapters run on dedicated endpoints. Fireworks has a preview called Specialized Model Inference that serves adapters on shared serverless GPUs at the base model's token price, but as of November 9, 2026 only for adapters trained on GLM 5.3 Flash (max rank 64, three adapters loaded per account). Everything else needs a dedicated deployment.

How much does it cost to host a fine-tuned model?

Mostly GPU-hours. On November 9, 2026: Fireworks on-demand H100 or H200 $8 per hour and B200 $13; Baseten H100 $0.10833 per minute (about $6.50 per hour) and H200 $0.125 per minute; Together AI dedicated inference H100 $5.49 per hour on demand; a self-managed RunPod H100 SXM pod $3.99 per hour. One GPU running all month (about 730 hours) costs roughly $2,900 to $5,800 before any scale-to-zero savings.

Should I merge my LoRA or serve it as an adapter?

Merge if you serve a single fine-tune: Fireworks recommends live merge because the merged model matches the base model's latency and throughput with no overhead. Serve adapters (multi-LoRA) when you have many variants of one base model, for example one per customer, because they share one deployment; the cost is some per-request overhead and lower peak throughput under heavy concurrency.

Quick Answer

How to Deploy a Fine-Tuned LLM in Production (2026)

Published: November 9, 2026

The short answer

Deploying a fine-tuned LLM in production comes down to two decisions: merge or adapter, then managed or self-hosted. As of November 9, 2026:

One fine-tune, want it managed: merge the LoRA into the base model and run it on a dedicated deployment — Fireworks live merge is the simplest path.
Many fine-tunes of one base model: serve them as adapters on one deployment — Fireworks multi-LoRA or Together AI dedicated endpoints with several adapters attached.
A full fine-tune or an unusual architecture: Baseten, which deploys any custom model and bills per minute.
Your own GPUs or cloud account: vLLM with LoRA enabled, behind a Kubernetes control plane (see self-hosting an OpenAI-compatible LLM with autoscaling).

Per-token serverless hosting for custom adapters has mostly disappeared: Together AI discontinued it, and Fireworks’ serverless preview covers only GLM 5.3 Flash adapters. Budget for GPU-hours.

The comparison

Prices from each vendor’s pricing page or documentation on November 9, 2026.

Host	How a fine-tune is served	GPU price	Idle cost	Best for
Fireworks	Live merge (one adapter, base-model speed) or multi-LoRA (many adapters); serverless LoRA preview for GLM 5.3 Flash only	H100 / H200 $8/hr, B200 $13/hr, B300 $15/hr, billed per GPU-second	Idle deployments scale to zero after one hour by default	Managed LoRA serving with the least setup
Together AI	Dedicated endpoints; several adapters on one base deployment for supported bases	Dedicated H100 $5.49/hr on demand	You pay while the endpoint runs	Teams already fine-tuning on Together
Baseten	Any custom model or merged checkpoint packaged with Truss	H100 $0.10833/min (~~$6.50/hr), H200 $0.125/min (~~$7.50/hr)	Per-minute billing with autoscaling replicas	Full fine-tunes, custom code paths
Self-hosted vLLM	Merged weights, or base model plus `--enable-lora` adapters	Whatever you rent: RunPod H100 SXM pod $3.99/hr, PCIe $2.89/hr	Yours to manage	Data residency, steady high volume

For a deeper ranking of hosts see best inference providers for LoRA and fine-tuned models and managed LoRA fine-tuning on dedicated GPUs.

Step 1 — Freeze an eval set before you touch infrastructure

Take 200–500 real prompts from the task the fine-tune exists for, plus the expected outputs or a grading rubric, and score both the base model and the fine-tune on them. This is the gate every later release must pass. Without it you cannot tell whether a new checkpoint, a quantised build or a new host made things worse. Wire it into CI: how to run LLM evals in CI/CD.

Step 2 — Merge or serve as an adapter

Fireworks’ documentation states the trade-off plainly. Live merge combines the LoRA weights with the base model at deployment time and produces a model that matches the base model’s latency, throughput and memory use — one adapter per deployment. Multi-LoRA loads adapters dynamically per request on one shared base deployment, which costs some per-request overhead and lowers maximum throughput under high concurrency, but lets dozens of variants share one set of GPUs.

Rule of thumb: one product, one fine-tune → merge. One adapter per customer, per language or per experiment → multi-LoRA.

Step 3 — Pick where it runs

Managed (Fireworks, Together AI, Baseten) gets you an endpoint in minutes and moves on-call to the vendor. Watch two things: the GPU-hour rate and whether idle deployments stop billing.
Self-hosted with vLLM (v0.32.0 was the latest stable release on October 18, 2026) gives you the same OpenAI-compatible API on your own hardware: vllm serve <base-model> --enable-lora --lora-modules support-bot=/models/support-lora serves the base model and the adapter side by side, and clients select the adapter by name in the model field.

Self-hosting only wins on cost at steady, high utilisation; do the arithmetic first with the self-host vs API breakeven guide.

Step 4 — Put a stable API in front

Expose an OpenAI-compatible endpoint whatever the host, and put a gateway in front for keys, rate limits, logging and fallback to a general model if the fine-tune’s endpoint is down. Your application should never call a vendor-specific deployment ID directly; swapping hosts then becomes a config change.

Step 5 — Roll out behind a canary

Send 5% of traffic to the new checkpoint, compare its eval-style scores and user signals against the current one for a few days, then ramp to 50% and 100%. Keep the previous checkpoint deployed until the new one has run at full traffic without regressions, so rollback is a routing change, not a redeploy.

Step 6 — Watch cost and drift

Track cost per thousand requests, p95 latency and the share of requests that fall back to the general model. Re-run the eval set every time the base model is updated or deprecated: an adapter trained on one base checkpoint does not transfer to the next, so a base-model retirement means a retrain.

Last verified: November 9, 2026.