How do I block AI shopping agents from my website?

Layer three controls. First, robots.txt: disallow the documented real-time agent user agents (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User), knowing several operators state robots.txt may not apply to user-initiated fetches. Second, the edge: in Cloudflare's AI bot policies set the Agent behaviour preset to Block (available on all plans since July 1, 2026) and, on Bot Management plans, write rules on cf.verified_bot_category and cf.bot_management.signed_agent; Vercel's WAF can match the Signature-Agent header. Third, terms of service: Amazon's September 20, 2026 block of Meta Muse relies on its Conditions of Use because the Ninth Circuit ruled on August 4, 2026 that the user, not the agent vendor, is the party accessing your systems under anti-hacking law.

Should merchants block AI shopping agents or allow them?

Most merchants should allow verified agents and block undeclared ones. Agents that identify themselves — via user-agent strings or, better, Web Bot Auth signatures — bring purchases with the user's intent already formed; Shopify chose to partner with Meta Muse on September 21, 2026 for exactly that reason. Amazon blocks them because agents bypass its $68 billion advertising business and it wants to own the customer relationship. If your revenue is the sale itself rather than on-site ads and upsells, an agent that buys is a customer. Block only agents that hide their identity, store credentials or ignore your robots.txt.

How can I tell if a visitor is an AI agent?

Three signals, in rising order of reliability. User-agent strings: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher and Amzn-User are documented, but any client can spoof them. Cloudflare classification: since July 1, 2026 Cloudflare labels bots by behaviour (Search, Agent, Training, Transact and others) and exposes Agent as a preset on every plan. Cryptographic signatures: agents using Web Bot Auth sign each request with an Ed25519 key published at /.well-known/http-message-signatures-directory and send Signature, Signature-Input and Signature-Agent headers; Cloudflare validates these and marks the request signed. Agents that drive a real browser and send no signal — as Amazon alleges Meta Muse does — are the hard case and need behavioural bot detection.

Does robots.txt stop AI shopping agents?

Only partly. Training crawlers such as GPTBot and ClaudeBot honour robots.txt, but real-time agents acting for a human are different: OpenAI says robots.txt rules may not apply to ChatGPT-User because a person requested the fetch, and Perplexity says the same about Perplexity-User. Agents piloting a full browser send no crawler token at all. Write the file anyway — it costs nothing, compliant operators respect it, and with Content Signals it is an express reservation of rights under Article 4 of EU Directive 2019/790 — but enforce your policy at the edge and in your terms.

Quick Answer

How to Control AI Shopping Agents on Your Website (2026)

Published: September 22, 2026

The short answer

Decide your policy first, then enforce it in three layers: robots.txt for declared crawlers, edge rules (Cloudflare Agent preset, Web Bot Auth) for real-time agents, and terms of service for the ones that hide. In September 2026 the two largest US commerce platforms picked opposite policies within 24 hours — Amazon blocked Meta’s Muse agent on September 20, Shopify partnered with it on September 21 — and both are rational. The right answer depends on whether you make money from the sale or from the browsing.

Step 1: Write the policy in one sentence per category

AI traffic is not one thing. Cloudflare formalised the split on July 1, 2026 by classifying bots by behaviour; the categories that matter to a store:

Category	Examples	What it does to a merchant	Typical decision
Search / answer indexing	OAI-SearchBot, PerplexityBot, Claude-SearchBot, meta-webindexer	Gets your products cited and linked in AI answers	Allow
Training	GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent	Absorbs catalogue text into model weights; no referral	Block or Disallow AI Training
Agent (real-time, user-directed)	ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User; Muse, Comet, Gemini Spark browsing	Reads pages or buys on a human’s behalf	Allow if declared; verify; rate-limit
Transact	Agent checkout flows (Link by Stripe, Shop Pay for agents)	Completes purchases	Allow with fraud controls, or partner

Write the line for each before touching a rule. “Block AI bots” as a single switch is how stores lose AI-answer traffic while still getting scraped.

Step 2: robots.txt — cheap, legal, partial

Add the documented tokens, exactly and case-sensitively:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

Content-signal: search=yes, ai-input=yes, ai-train=no

Three facts to know:

Real-time agents may ignore it by design. OpenAI states robots.txt “may not apply” to ChatGPT-User; Perplexity says the same for Perplexity-User. A human asked, so they fetch.
Browser-driving agents send nothing. Muse’s Chromium, Comet and ChatGPT’s visual browser present as a browser. Amazon’s complaint that Muse “doesn’t identify itself” is this problem exactly.
It still has legal weight. Content Signals restrictions are express reservations of rights under Article 4 of EU Directive 2019/790, and a robots.txt an agent ignored strengthens a terms-of-service case.

Keep /robots.txt returning 200 — a security rule that blocks it silently disables your whole policy.

Step 3: Edge enforcement

Cloudflare (all plans). Security → AI bot policies exposes three presets: Search, Agent, Training, each set to Allow, Block on all pages, or Block on pages with ads. Set Agent deliberately. Since September 15, 2026 newly onboarded domains default to Training and Agent blocked on pages with ads — so a store that “never configured anything” may already be blocking agents it wants. AI Crawl Control (free) lists every crawler seen, its category, robots.txt violations and a per-crawler allow/block.

Cloudflare Bot Management (paid). Use cf.verified_bot_category to allow Search and Agent while blocking Training, and cf.bot_management.signed_agent to allow only agents that cryptographically identify themselves. Cloudflare labels verified bots Direct (one operator, own infrastructure) or Intermediary (a service many end users drive — the Muse/ChatGPT case), and is experimenting with forwarding end-user information in the RFC 7239 Forwarded header so you can apply policy to the actual person.

Vercel. WAF custom rules can match the User-Agent or the Signature-Agent header to allow or block declared agents.

Rate limits. Audit old anti-scraper rules: a limit written in 2024 will throttle OAI-SearchBot and Amzn-SearchBot today and cost you AI-answer placements.

Step 4: Verify identity with Web Bot Auth

User-agent strings are claims; signatures are proof. Under Web Bot Auth an agent generates an Ed25519 key pair, publishes the public key as a JWKS at /.well-known/http-message-signatures-directory, registers it with Cloudflare, and signs each request with HTTP Message Signatures (Signature, Signature-Input, Signature-Agent headers). Signed agents have counted as verified bots since July 1, 2026, and an agent that abuses trust loses verified status. As of August 2026, chatgpt.com and browserbase.com served keys at that path; most operators did not yet. Policy shifts from “which string is in the header” to “who signed this and on whose behalf.” Details: What is Web Bot Auth?

Step 5: Terms of service — the layer that survived court

Amazon won an injunction against Perplexity’s Comet in March 2026, then lost it on August 4, 2026 when the Ninth Circuit held that under federal anti-hacking law the user is the party accessing Amazon’s computers, not the agent vendor; rehearing was denied September 10. Contract claims survived. That is why Amazon’s Muse block message reads “Continued access by an unauthorized AI agent violates Amazon’s Conditions of Use, to which our customers have agreed” — addressed to customers, grounded in terms.

If you intend to block, update your terms to (a) define automated agents acting for a user, (b) require them to identify themselves and honour robots.txt, and (c) reserve the right to refuse undeclared agents. If you intend to allow, publish an agent policy page saying so — declared agents’ operators read them.

Step 6: If you allow — make it work for you

Structured data. Product, Offer, price, availability and shipping schema are what agents parse; a page an agent cannot read is a sale it makes elsewhere.
Agent-friendly checkout. Shopify’s Muse partnership and Shop Pay for agents, Link by Stripe’s one-time cards with purchase protection, and Visa’s agent trust framework are the emerging rails. Accepting one-time virtual cards avoids the credential-storage fight Amazon is having.
Log agent share. Segment sessions and conversions by Cloudflare category or signed-agent flag. Muse alone had 2.5 million installs by September 21, 2026; if your competitors accept agent checkout and you do not, you will see it here first.
Fraud controls unchanged. Treat agent orders like any card-not-present order; the human’s intent is real, the card is often virtual.

Decision matrix

Your revenue comes from	Undeclared browser agents	Declared / signed agents	Training crawlers
The sale (DTC, Shopify store)	Rate-limit, watch	Allow, optimise	Block
On-site ads, sponsored placements (marketplace)	Block via terms + edge	Allow only via partnership	Block
Content / affiliate	Block	Allow Search, consider Agent	Block