Three kinds of AI traffic
AI companies send three different kinds of request, and they matter for different reasons:
- Fetches for a person. Someone asked an assistant about your page, and it fetched it then. This is the clearest sign of real interest. Operators say these agents may not follow robots.txt, because a person asked.
- AI search crawlers. They build the index an assistant searches. Blocking them can keep you out of AI answers.
- Training crawlers. They collect content that may be used to train models. Blocking them doesn't affect whether assistants can fetch or cite you.
The agents
Each row comes from the operator's own documentation, read on 23 October 2026. "Follows robots.txt" is what the operator says.
| Agent | Operator | Kind | Follows robots.txt | Published IP list |
|---|---|---|---|---|
ChatGPT-User | OpenAI | For a person | May not | chatgpt-user.json |
OAI-SearchBot | OpenAI | AI search | Yes | searchbot.json |
GPTBot | OpenAI | Training | Yes | gptbot.json |
Perplexity-User | Perplexity | For a person | May not | perplexity-user.json |
PerplexityBot | Perplexity | AI search | Yes | perplexitybot.json |
Claude-User | Anthropic | For a person | Yes | bots.json |
Claude-SearchBot | Anthropic | AI search | Yes | bots.json |
ClaudeBot | Anthropic | Training | Yes | bots.json |
Google-Agent | For a person | May not | user-triggered-agents.json | |
Googlebot | Search engine | Yes | common-crawlers.json | |
Google-Extended | robots.txt token only | (a token) | None: no crawler sends it | |
Bingbot | Microsoft | Search engine | Yes | bingbot.json |
DuckAssistBot | DuckDuckGo | For a person | Yes | duckassistbot.json |
Applebot | Apple | Search engine | Yes | applebot.json |
Applebot-Extended | Apple | robots.txt token only | (a token) | None |
Amzn-User | Amazon | For a person | May not | Not published |
Amzn-SearchBot | Amazon | AI search | Yes | Not published |
Amazonbot | Amazon | Training | Yes | Not published |
Meta-ExternalFetcher | Meta | For a person | May not | Not published |
Meta-WebIndexer | Meta | AI search | Yes | Not published |
Meta-ExternalAgent | Meta | Training | Yes | Not published |
CCBot | Common Crawl | Training | Yes | ccbot.json |
Sources: OpenAI, Perplexity, Anthropic, Google's common crawlers and user-triggered fetchers, Microsoft, DuckDuckGo, Apple, Amazon, Meta, Common Crawl.
Google-Extended and Applebot-Extended are not crawlers
They are names you can use in robots.txt to say whether your content may be used for training Google's and Apple's models. No request ever carries them, so you won't find them in your logs. Apple notes that disallowing Applebot-Extended still lets Applebot crawl your pages for Spotlight, Siri and Safari.
Verify before you trust a user agent
A user agent is just text; anyone can send GPTBot. A request is genuine only if its IP address is in the list its operator publishes. For Google, the verification guide also describes a reverse DNS check: the address should resolve to a googlebot.com, google.com or googleusercontent.com name that resolves back to the same address.
When the operator publishes no list, as with Amazon and Meta, a request can only be claimed, never verified.
Check yours
- See which of these agents your robots.txt allows with the free AI crawler checker.
- See which actually visit, and which are impostors, with AI visibility, which checks every request against these lists, refreshed daily.