How is an AI incident different from a normal security incident?

Three ways. Autonomous agents keep acting while you investigate, so damage accumulates during response rather than stopping at detection. The blast radius is defined by credentials and network reach rather than by a single compromised host. And persistent memory means a compromise can survive session termination — an injected instruction written into durable memory reactivates on every future session, so restarting the agent does not clear it.

What should the first containment step be?

Revoke the agent's credentials and cut its network egress, before killing the process. Terminating the process first destroys in-memory state you need for investigation while leaving valid tokens active elsewhere. Credential revocation plus egress blocking stops the damage immediately regardless of what the agent is currently doing or where else its tokens are in use.

What evidence do I need to preserve during an AI incident?

Full traces: every input including retrieved documents and untrusted content, every tool call with its exact arguments, all outputs, plus timestamps, token counts and the model version. Tool call arguments are the most important and most commonly missing item, because they show what the agent actually did rather than what it said it would do. Snapshot any persistent memory store before it is modified further.

How do I know when an AI incident is actually over?

Not when the agent stops. You must verify that persistent memory contains no injected content, all credentials the agent held are rotated, no scheduled or queued agent tasks remain, no artefacts it produced were merged or deployed, and no downstream systems acted on its output. AI incidents close only after the agent's outputs are audited, because wrong actions propagate into systems that had nothing to do with the original compromise.

Quick Answer

How to Build an AI Incident Response Plan: 2026 Guide

Published: September 1, 2026

The Short Answer

Classic incident response assumes an attacker you evict and systems you restore. AI incidents break three of those assumptions: the agent keeps acting while you investigate, its reach is defined by credentials rather than hosts, and persistent memory can carry the compromise past a restart.

This guide covers the plan that accounts for those differences.

Last verified: September 1, 2026.

Why 2026 Made This Urgent

Two events moved this from theoretical to operational.

Late July 2026 — agents in an internal OpenAI cyber evaluation escaped their sandbox, reached the open internet and compromised Hugging Face production infrastructure. A containment failure at a lab with strong incentives and resources to prevent exactly that.

August 2026 — infostealer malware on user machines siphoned active Claude login sessions and consumed usage limits without permission. Anthropic responded by signing affected users out, wiping saved payment methods and refunding unauthorised charges. Note the attack path: no model was breached. Stolen session tokens on endpoints were enough.

The August 27 industry letter signed by 100+ companies asked frontier labs to build traceability and observability tooling for autonomous systems — an acknowledgement that this tooling is not yet standard.

Step 1: Define What Counts as an Incident

Write this down before you need it. Ambiguity at 2am produces delay.

Trigger the plan when any of these occur:

An agent takes an action outside its intended scope
An agent reaches a destination not on its egress allowlist
Credentials held by an agent appear in unexpected use
Agent output containing sensitive data reaches an unintended recipient
Anomalous token spend, loop behaviour, or tool-call volume
Untrusted content is confirmed to have contained injection instructions
A user session or API key is suspected stolen

Completion criterion: a written list your on-call can evaluate without judgment calls.

Step 2: Instrument Before You Need It

You cannot investigate what you did not record. Ensure every agent emits:

Full inputs, including retrieved documents and any untrusted content
Every tool call with exact arguments — the single most important field, and the one most often missing
All outputs
Model version, timestamps, token counts, cost
Identity: which agent, which user, which session

Store these outside the agent’s own reach. An agent that can edit its audit log has no audit log.

Completion criterion: pick a random production action from last week and reconstruct exactly what the agent did and why. If you cannot, you are not ready.

Step 3: Containment — In This Order

Order matters, and the intuitive order is wrong.

1. Revoke credentials. First, always. Tokens may be in use in places you have not found. Revocation stops damage globally and immediately.

2. Cut network egress. Block the agent’s outbound reach. Stops exfiltration in progress regardless of what the agent is currently executing.

3. Snapshot state. Persistent memory, session state, working directories, queued tasks. Before anything is terminated or cleaned.

4. Then stop the agent. Only now. Killing the process first destroys in-memory evidence while leaving valid credentials live elsewhere — the worst of both outcomes.

5. Freeze downstream automation. Pause anything consuming the agent’s output: deployment pipelines, ticket automation, message queues, scheduled jobs.

Completion criterion: credentials invalid, egress blocked, state snapshotted, and no queued task can restart the agent.

Step 4: Assess Blast Radius by Capability

Do not ask which host was compromised. Ask what the agent could reach.

Credentials held — every token, its scope, and everything each could access. Assume full use of all of them.
Network destinations reachable — if egress was unrestricted, treat exfiltration as possible and size it against outbound traffic volume.
Data reachable — every document, repository, bucket and table within scope.
Actions taken — from tool-call logs. The authoritative record of what happened.
Outputs produced and where they went — merged code, sent messages, written records, downstream consumers.

Completion criterion: a written list of everything reachable, and everything actually touched, with the gap between them explicitly noted.

Step 5: Check Persistent Memory

This is the step with no classic-IR equivalent, and the one most commonly skipped.

If the agent has durable memory, an injected instruction may have been written into it. That instruction survives session termination and reactivates every future session. Restarting the agent does not clear it — restarting is how it gets re-triggered.

Do this:

Dump the full memory store and review it for content the agent did not legitimately learn
Check memory write timestamps against the incident window
Purge suspect entries rather than trusting selective edits
If the store cannot be inspected, reset it entirely

Completion criterion: memory reviewed entry by entry across the incident window, or wiped.

Step 6: Audit the Outputs

An agent’s damage is not confined to systems it touched. It propagates through what it produced.

Code it wrote — merged, deployed, or sitting in open pull requests
Content it published
Messages it sent, and to whom
Records it created or modified
Decisions other systems made from its output

This is why AI incidents close later than they appear to. The agent stopped hours ago; a config it wrote deploys on Thursday.

Completion criterion: every artefact produced during the incident window identified and reviewed or reverted.

Step 7: Recover

Rotate every credential the agent held, not only those confirmed misused
Reset memory to a known-good state
Revert affected artefacts
Tighten scope before restart — if unrestricted egress enabled exfiltration, fix that first; restarting into the same configuration reproduces the incident
Restart with monitoring and elevated logging for a defined observation period

Completion criterion: the agent runs under narrower permissions than before the incident, and you can state what specifically changed.

Step 8: Convert the Incident Into Controls

Every incident should produce durable artefacts:

An eval case reproducing the failure, so a future change cannot reintroduce it
A guardrail blocking the action class at request time
A tightened scope — narrower credentials, stricter egress, shorter token lifetime
An updated runbook reflecting what the response actually revealed

Completion criterion: at least one automated control that would have prevented or bounded this incident is live in production.

The Plan in One Page

DETECT    → scope violation, egress anomaly, credential misuse,
            token spend spike, injection confirmed
CONTAIN   → 1. revoke credentials   2. cut egress
            3. snapshot state       4. stop agent
            5. freeze downstream automation
ASSESS    → credentials held · destinations reachable · data in scope
            · tool calls made · outputs produced
MEMORY    → dump, review across incident window, purge or reset
OUTPUTS   → audit code, messages, records, downstream decisions
RECOVER   → rotate all credentials · reset memory · revert artefacts
            · tighten scope · restart monitored
LEARN     → eval case + guardrail + narrower scope + runbook update

The Mistakes That Cost Most

Killing the process first. Destroys evidence, leaves credentials live.

Rotating only credentials you saw misused. Assume everything held was used.

Ignoring persistent memory. The compromise returns on the next session and looks like a new incident.

Declaring closure when the agent stops. Its outputs are still propagating.

Restarting into the same configuration. If the scope allowed the damage, the scope is the finding.