How do I block AI crawlers in robots.txt?

Add a group per user agent with a Disallow rule. For example, "User-agent: GPTBot" followed by "Disallow: /" blocks OpenAI's training crawler from the whole site. Each AI operator uses distinct tokens for training, live retrieval, and search indexing, so blocking one does not block the others — you must name each agent you intend to affect.

Which robots.txt group applies when a crawler matches more than one?

Crawlers follow the single most specific matching user-agent group and ignore all others, including the wildcard group. If your file contains both "User-agent: *" and "User-agent: GPTBot", GPTBot obeys only the GPTBot group. Rules do not merge, so any directive you want to apply to a named agent must be repeated inside its own group.

Does robots.txt legally prevent AI training on my content?

No. robots.txt is a voluntary protocol with no enforcement mechanism. Major operators publicly honour it, but a crawler that ignores it faces no technical barrier. If you need enforcement rather than a request, you need edge rules or WAF policies that inspect and block requests, backed by IP verification.

Blog Crawlers

robots.txt for AI crawlers: a directive-by-directive reference

Every AI user agent worth naming, what each one controls, the precedence rules that decide which directive wins, and the wildcard mistake that quietly removes sites from AI answers.

July 11, 2026 · Updated August 7, 2026 Markdown for agents

robots.txt is thirty years old and was never designed for the question we now ask of it: not “may you index this” but “may you learn from this, and may you read it aloud to someone”. Operators have adapted by issuing separate tokens per purpose. The result is a file that has to be written more carefully than it used to be.

The agents

OpenAI

Token	Purpose
`GPTBot`	Bulk crawl for model training
`ChatGPT-User`	Live fetch during a user’s conversation
`OAI-SearchBot`	Index for ChatGPT search

Anthropic

Token	Purpose
`ClaudeBot`	Bulk crawl for model training
`Claude-User`	Live fetch during a user’s conversation
`Claude-SearchBot`	Index for Claude search

Google

Token	Purpose
`Googlebot`	Search index; also feeds AI Overviews
`Google-Extended`	Not a crawler — a control token for Gemini training use

Google-Extended is the one people get wrong. It will never appear in your logs, because no request is ever made under that name. It is a switch that governs how content Googlebot already fetched may be used for Gemini. Disallowing it does not affect your search ranking, and it does not remove you from AI Overviews, which are served from the search index.

Others

Token	Operator	Purpose
`PerplexityBot`	Perplexity	Index
`Perplexity-User`	Perplexity	Live fetch during an answer
`Applebot`	Apple	Siri and Spotlight
`Applebot-Extended`	Apple	Control token for Apple model training
`CCBot`	Common Crawl	Public archive used by many training datasets
`Bytespider`	ByteDance	Training crawl; widely reported to be aggressive
`meta-externalagent`	Meta	Training crawl
`Amazonbot`	Amazon	Alexa and index

CCBot deserves attention out of proportion to its traffic. Common Crawl is a nonprofit archive, and a great many training datasets are derived from it. Allowing CCBot while blocking GPTBot is a coherent position, but it is not the position most people think they are taking.

The precedence rule that breaks files

A crawler obeys exactly one group: the most specific user-agent match. Groups do not merge.

This one rule causes most broken robots.txt files. Consider:

User-agent: *
Disallow: /admin/
Crawl-delay: 10

User-agent: GPTBot
Disallow: /pricing

GPTBot here is allowed to crawl /admin/. It matched its own group, so the wildcard group — including the /admin/ disallow — does not apply to it at all. Anything you want a named agent to obey has to be restated inside that agent’s group:

User-agent: GPTBot
Disallow: /admin/
Disallow: /pricing

Within a group, the most specific path rule wins regardless of order, and Allow beats Disallow on ties. So this permits /blog/ while blocking everything else:

User-agent: GPTBot
Disallow: /
Allow: /blog/

Other mechanics worth knowing: * matches any sequence and $ anchors the end, so Disallow: /*.pdf$ blocks PDFs. Matching is case-sensitive on paths and case-insensitive on user-agent tokens. And robots.txt is per host and per scheme — example.com and www.example.com need their own files unless one redirects to the other.

Three configurations

Open, with the application fenced off. The default for most B2B software. Marketing and docs are the point; the app is not.

User-agent: *
Allow: /
Disallow: /app/
Disallow: /api/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

No training, yes retrieval. For publishers who want citations but not corpus inclusion.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Training on the archive, not on premium. For a site with both free and paid content.

User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /research/
Disallow: /members/

The mistake that removes you from AI answers

Someone reads a headline about AI scraping, adds a broad block, and ships it:

User-agent: *
Disallow: /

That does block GPTBot. It also blocks ChatGPT-User, Perplexity-User, and Claude-User — the agents that fetch your page because a real person asked a question you could answer. The company has not protected an asset; it has opted out of its own category’s AI answers, and because nothing in its analytics ever showed that traffic, nobody notices for two quarters.

If you take one thing from this: decide about training and about retrieval separately, and write the file so those decisions are visible.

Where robots.txt stops

It is a request, not a fence. There is no enforcement, and compliance is a policy choice by each operator. Well-behaved crawlers honour it; scrapers impersonating well-behaved crawlers do not, and the user-agent string they send costs them nothing to fake.

Enforcement means inspecting requests where you can actually reject them: WAF rules, edge middleware, or a proxy in front of your origin, matching on verified IP ranges rather than the user-agent header alone. Shiftrank does this at the edge and — more usefully at first — runs in observe mode, so you can see exactly which agents would have been blocked before any rule takes effect. Getting that order right matters, because a robots.txt mistake is silent and a blocking-rule mistake is not.

Common questions

How do I block AI crawlers in robots.txt?: Add a group per user agent with a Disallow rule. For example, "User-agent: GPTBot" followed by "Disallow: /" blocks OpenAI's training crawler from the whole site. Each AI operator uses distinct tokens for training, live retrieval, and search indexing, so blocking one does not block the others — you must name each agent you intend to affect.
Which robots.txt group applies when a crawler matches more than one?: Crawlers follow the single most specific matching user-agent group and ignore all others, including the wildcard group. If your file contains both "User-agent: *" and "User-agent: GPTBot", GPTBot obeys only the GPTBot group. Rules do not merge, so any directive you want to apply to a named agent must be repeated inside its own group.
Does robots.txt legally prevent AI training on my content?: No. robots.txt is a voluntary protocol with no enforcement mechanism. Major operators publicly honour it, but a crawler that ignores it faces no technical barrier. If you need enforcement rather than a request, you need edge rules or WAF policies that inspect and block requests, backed by IP verification.

Keep reading

GPTBot vs ChatGPT-User vs OAI-SearchBot: which one is reading your site
OpenAI operates three distinct crawlers with three different purposes. Blocking the wrong one costs you citations; allowing the wrong one costs you leverage. Here is how to tell them apart in your logs.

See which answer engines already read this page

Shiftrank records every AI crawler hit from the moment your DNS resolves.

Start free