robots.txt checker and tester
The crawler's product token, such as Googlebot, GPTBot or ClaudeBot.
Allowed
Decided by line 3: Allow: /admin/public (from the * group: no group names this agent)
What crawlers will see
2 groups, 3 rules, 0 sitemaps
No lines that crawlers would ignore.
How it works
Rules are read the way RFC 9309 defines: fields are case-insensitive, consecutive user-agent lines share the rules below them, and a crawler merges every group that names its token, or falls back to the * group.
For a URL, the longest matching allow or disallow path wins; when an allow and a disallow match with the same length, allow wins. * matches any characters and $ anchors the end of the URL. /robots.txt itself is always allowed.
The checker also lists lines that crawlers will ignore: rules before any user-agent line, unknown fields such as Noindex, and lines without a colon.
Questions
Is robots.txt case-sensitive?
Field names and user-agent tokens are not. Paths are: Disallow: /Admin does not block /admin.
Does Crawl-delay work?
It is not part of RFC 9309. Anthropic says its crawlers support it; Google's robots.txt documentation says it does not.
Read next
- AI crawler checkerFree AI crawler checker: enter your site and see whether GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and 18 other crawlers may read it, per robots.txt.
- What is robots.txt?robots.txt: a text file at the root of a website that tells crawlers which URLs they may fetch. A plain-language definition from the LogZuno glossary.
- AI crawler analytics, from your server logs to a reportAI crawler analytics in practice: find ChatGPT, Perplexity and Claude in your server logs, verify them against their IP lists and measure what they read.
Want to see which AI crawlers actually visit?
7 days free on any plan, up to 500 events. Then from $17 a month. Cancel during the trial and you pay nothing.