AI crawler checker
How it works
The checker downloads your site's /robots.txt and applies the matching rules from RFC 9309, the Robots Exclusion Protocol, for each crawler: a crawler follows the group that names its token, otherwise the * group; the longest matching path wins, and allow wins a tie.
The crawler list comes from each operator's own documentation. Some agents that fetch a page because a person asked (ChatGPT-User, Perplexity-User, Google-Agent, Amzn-User, Meta-ExternalFetcher) may not follow robots.txt, according to their operators, so for them the result shows what you asked for, not a guarantee.
robots.txt only states a preference. To see which of these crawlers actually visit, and whether a visitor claiming to be GPTBot really comes from OpenAI, you need your server's own traffic, which is what LogZuno's AI visibility reads.
Questions
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI uses GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User when a person asks ChatGPT to open a page. Blocking GPTBot does not block the other two.
What does the checker do if robots.txt is missing?
Under RFC 9309, a 404 or other 4xx response means crawlers may fetch everything. A 5xx error, a 429 or no answer means they should assume everything is disallowed until the file can be read.
Read next
- robots.txt checker and testerFree robots.txt checker: fetch or paste a robots.txt, test any URL against any user agent, and see the exact rule that allows or blocks it, per RFC 9309.
- AI crawler user agents, what they do and how to verify themAI crawler user agents in 2026: every agent from OpenAI, Perplexity, Anthropic, Google, Apple, Amazon, Meta and Common Crawl, its job and its IP list.
- AI visibility: what AI assistants read, and who they send youAI visibility from your own server traffic: which pages ChatGPT, Perplexity, Claude and Google read, checked against their IP lists, and who they send.
Want to see which AI crawlers actually visit?
7 days free on any plan, up to 500 events. Then from $17 a month. Cancel during the trial and you pay nothing.