What infrastructure is needed to run AI agents at scale?

Seven layers, as of October 2026: (1) model access through a gateway with fallbacks and budgets; (2) durable execution so long agent runs survive crashes and retries, for example Temporal Cloud from $50 per million actions; (3) isolated sandboxes for code and browser actions, for example E2B (Pro $150 a month plus per-second compute); (4) tool and agent protocols, MCP for agent-to-tool and A2A for agent-to-agent; (5) per-agent identity and secrets brokering; (6) tracing and evaluation, for example Langfuse (Core $29 a month) on OpenTelemetry GenAI conventions; and (7) policy and human approval gates for risky actions.

What is the difference between MCP and A2A for agent infrastructure?

MCP (Model Context Protocol) connects an agent to tools, data and applications: client-server, JSON-RPC, with capability discovery. A2A (Agent2Agent) connects independent agents to each other so they can discover one another, exchange tasks and delegate work, including long-running asynchronous tasks. MCP was contributed to the Linux Foundation's Agentic AI Foundation on December 9, 2025; A2A is a Linux Foundation project that passed 150 supporting organisations by April 9, 2026.

What should you monitor for AI agents in production?

Trace every run end to end: each model call, tool call and handoff with latency, tokens and cost; retries and loops; tool error rates; retrieval quality; and policy violations or blocked actions. Add offline evaluations on a fixed task set before each release and online scoring on sampled production runs. Use OpenTelemetry's generative-AI semantic conventions so traces are portable between vendors.

Do I need Kubernetes to run AI agents at scale?

No. Most teams run the agent loop as ordinary services or workflow workers and rent the hard parts: a model gateway, managed durable execution, hosted sandboxes and a tracing service. Kubernetes becomes worth it when you self-host models or need your own sandbox fleet for data-residency reasons.

Quick Answer

What Infrastructure Do You Need to Run AI Agents at Scale?

Published: October 17, 2026

The short answer

Running agents at scale is less about the model and more about seven pieces of plumbing around it: a model gateway, durable execution, sandboxes, protocols, identity, observability and approval gates. A single agent in a notebook needs none of them; a thousand concurrent agents that take actions on real systems need all of them. Tool names and prices below are as of October 17, 2026.

The seven layers

Layer	What it does	Example tools (Oct 2026)	Why it matters at scale
1. Model gateway	One API over many models; fallbacks, rate limits, budgets, caching	LiteLLM, OpenRouter, Cloudflare AI Gateway	Provider outages and rate limits stop fleets, not single agents
2. Durable execution	Persists each step so runs resume after crashes, deploys and timeouts	Temporal Cloud ($50 per 1M actions, $150 free credits), Inngest, Restate	Agent runs last minutes to days; retries must not repeat side effects
3. Sandboxes	Isolated machines for code, shell and browser actions	E2B (Hobby free with $100 credit; Pro $150/mo + per-second usage; 100 concurrent on Pro), Modal, Daytona	Untrusted generated code must not touch production hosts
4. Protocols	Standard tool and agent interfaces	MCP (agent→tool), A2A (agent→agent)	Avoids N×M custom integrations
5. Identity and secrets	One identity per agent, short-lived credentials, brokered secrets	Infisical, HashiCorp Vault, AWS Secrets Manager	A prompt injection can only steal what the agent can read
6. Observability and evals	Traces of every model and tool call; offline and online evaluation	Langfuse (Core $29/mo, Pro $199/mo), Arize Phoenix, LangSmith; OpenTelemetry GenAI conventions	You cannot debug a 40-step run from logs
7. Policy and approvals	Allow/deny rules, spend caps, human sign-off on risky actions	Gateway policies, workflow signals, approval queues	Limits the blast radius of a wrong action

How the pieces fit

A production agent run looks like this: a workflow (layer 2) starts the run and records each step; each step calls a model through the gateway (1) and tools over MCP (4); code and browsing happen inside a sandbox (3) holding only short-lived credentials (5); every call emits a trace (6); and any action above a risk threshold pauses for an approval signal (7) before the workflow continues. Hand-offs to another team’s agent go over A2A.

Protocols: MCP vs A2A

MCP is client-server over JSON-RPC: the agent discovers a server’s tools, resources and prompts and calls them. Anthropic contributed it to the Linux Foundation’s Agentic AI Foundation, formed December 9, 2025 with Block’s goose and OpenAI’s AGENTS.md.
A2A lets independent agents discover each other through agent cards, exchange tasks and stream results for long-running, asynchronous work. The Linux Foundation reported more than 150 supporting organisations and integration across Google, Microsoft and AWS on April 9, 2026.

Use MCP inside your system; add A2A only where agents owned by different teams or vendors must collaborate.

What to measure

Borrow from distributed-systems monitoring, then add AI-specific signals:

Per run: latency, steps, tokens and cost; retries and loop detection; final success or failure.
Per tool: call volume, error rate, p95 latency.
Quality: task success on an offline eval set before each release; sampled online scoring (LLM-as-judge plus human review).
Safety: blocked actions, policy violations, approval wait times, secrets accessed per agent.

Emit traces using OpenTelemetry’s generative-AI semantic conventions so you can switch tracing vendors without re-instrumenting.

A build order that works

Tracing first — you will need it to debug everything else.
Gateway with a fallback model and per-agent budgets.
Durable execution once runs exceed a few minutes or call paid or irreversible APIs.
Sandboxes before any agent executes generated code.
Per-agent identity and brokered secrets before agents touch production systems.
Approval gates on writes, payments, emails and deletes.
A2A only when you have agents across team or company boundaries.

Go deeper: best LLM gateways, best durable execution platforms for AI agents, best AI agent sandboxes, best secrets managers for AI agents, how to monitor and evaluate AI agents in production, MCP vs A2A vs ACP.

Last verified: October 17, 2026. Prices are list USD from vendor pricing pages.