Is Baseten good for batch processing?

Yes for custom or fine-tuned models, with limits. As of November 4, 2026, Baseten's async inference queues requests to any dedicated deployment without code changes, holds them for up to 72 hours (default 10 minutes, set by max_time_in_queue_seconds), runs each attempt for up to 1 hour with retries, and delivers results by webhook. It does not work on Baseten Model APIs, cannot stream, and Baseten does not store outputs, so a failed webhook loses the result. You pay the normal per-minute replica price; there is no batch discount.

Does Baseten offer spot or preemptible GPUs?

Not on its public price list. Baseten's instance-type reference on November 4, 2026 lists one on-demand per-minute price per instance (for example H100 $0.10833 a minute, about $6.50 an hour) and no spot tier. Discounts come from Pro and Enterprise commitments. For interruptible GPU capacity, Modal's GPU functions are preemptible by default at its list price (H100 about $3.95 an hour), and RunPod's Community Cloud is cheaper than its Secure Cloud.

What is the cheapest way to run a large LLM batch job?

If the model is available through a provider Batch API, use that: OpenAI, Anthropic and Google Gemini all charge 50% of the standard price for batch requests with a 24-hour completion window. Rent GPUs (Baseten, Modal, RunPod) only when the model is custom, fine-tuned or not offered by a batch API, and then pick per-second billing (Modal, RunPod) over per-minute billing for short jobs.

Does Together AI's Batch API give 50% off every model?

No. Together's batch documentation says jobs run at up to 50% off serverless rates, but on November 4, 2026 its discounted-model list contains only openai/whisper-large-v3; other models run at standard rates. Batch still gives a separate rate-limit pool, up to 50,000 requests per batch and no charge for failed requests.

Quick Answer

Baseten for Batch Processing and Spot GPUs: An Evaluation

Published: November 4, 2026

The short answer

Baseten handles batch work well for models you deploy yourself, but it has no batch discount and no spot GPUs. Its async inference queues requests to any dedicated deployment for up to 72 hours and posts results to your webhook, billed at the normal per-minute replica price. For the cheapest batch run:

Model available through a provider Batch API? Use it — OpenAI, Anthropic and Gemini charge 50% of standard price.
Custom or fine-tuned model, spiky jobs? Modal — per-second billing, preemptible GPUs, scale to zero.
Custom model, steady overnight volume, you want managed serving? Baseten async inference.
Lowest raw GPU price and you run the stack? RunPod Community Cloud pods.

This page is a spoke of our Baseten evaluation. For list GPU prices across platforms see Baseten pricing compared to other ML infrastructure tools.

Baseten async inference: what it does

Property	Baseten async inference (Nov 4, 2026)
Works on	Any dedicated deployment, no code changes
Not available on	Model APIs (per-token endpoints)
Max time in queue	Default 10 minutes, up to 72 hours (`max_time_in_queue_seconds`)
Max time per attempt	1 hour, with a configurable retry policy
Priority	`priority` 0, 1 or 2 (0 runs first; default 0)
Result delivery	Webhook to your HTTPS endpoint, with signature verification
Output storage	None — if every webhook delivery fails, the output is lost
Streaming	Not supported
Price	Normal per-minute replica billing; no batch discount

The queue is decoupled from scaling: requests can sit in the queue while the deployment has zero replicas, and the queue processor’s calls trigger the autoscaler. That makes a nightly job cheap to keep idle all day — set min_replica: 0 and let the queue wake the model.

Two operational rules: persist outputs yourself (in the webhook receiver or from the model’s postprocess()), and check the errors array in each payload before accepting data.

Spot GPUs: Baseten does not sell them

Baseten’s instance reference lists one on-demand price per instance — H100 $0.10833/minute (about $6.50/hour), H200 $0.125 ($7.50), B200 $0.16633 (about $9.98), A100 $0.06667 (about $4.00) — and no spot or preemptible tier. Lower prices come from Pro and Enterprise contracts, and Enterprise can run in your own cloud, where you could use your provider’s committed spend.

If interruptible capacity is the point:

Modal — every function is preemptible by default and restarts on the same input after a preemption; GPU functions cannot be made non-preemptible. Its list price already reflects that (H100 $0.001097/second, about $3.95/hour). Checkpoint long jobs.
RunPod — Community Cloud pods are cheaper than Secure Cloud (H100 SXM $2.69 vs $3.99/hour on October 28, 2026).

Provider Batch APIs: the 50% option

API	Discount	Completion window	Limits
OpenAI Batch	50%	Within 24 hours	Separate, higher rate-limit pool; 200 MB input file; one model per file
Anthropic Message Batches	50%	Most within 1 hour; expire after 24 hours	100,000 requests or 256 MB per batch; results kept 29 days; not ZDR-eligible
Gemini Batch	50%	Target 24 hours	2 GB input file
Together Batch	Up to 50% — only `whisper-large-v3` listed as discounted	24-hour best effort	50,000 requests, 100 MB file; failed requests not billed; files kept 7 days

With batch pricing, Claude Sonnet 5.5 drops from $2/$10 to $1/$5 per million tokens and GPT-6.1 Terra from $1/$5 to $0.50/$2.50 — see current API prices. For most evaluation, classification and enrichment jobs that is cheaper than any rented GPU, with no serving to run.

When Baseten is the right batch platform

Your own model: a fine-tune, a custom vision or audio pipeline, or an open model with a custom engine — none of which a provider Batch API can run.
Mixed traffic: the same deployment serves real-time requests by day and async batch by night; async requests compete with sync requests through predict_concurrency and priority.
Data stays in your cloud: Baseten Enterprise self-hosted deployments.

When it is not

Short or one-off jobs: per-minute billing and paid cold starts favour Modal’s per-second meter.
Popular open or closed models: a 50%-off Batch API beats any GPU rental on price and effort.
Interruptible workloads that can checkpoint: Modal or RunPod Community Cloud cost less per GPU-hour.

How to evaluate in a day

Run 1,000 representative requests through a provider Batch API (if the model exists there) and record cost per request.
Deploy the same model on Baseten with min_replica: 0, submit the same 1,000 through /async_predict, and read the cost from baseten org billing usage.
Repeat on Modal with the same engine.
Compare cost per successful output, including cold starts and failed attempts.

Last verified: November 4, 2026.