Definition
robots.txt is a plain-text file at /robots.txt that tells crawlers which parts of a site they may fetch. Its format is standardised in RFC 9309, the Robots Exclusion Protocol.
User-agent: *
Disallow: /admin
User-agent: GPTBot
Disallow: /How crawlers read it
- A crawler follows the group that names its user agent, otherwise the
*group. - The longest matching rule wins; on a tie,
Allowwins. *matches any characters and$anchors the end of a path.
What it doesn't do
It's a request, not a lock: well-behaved crawlers follow it, others don't. It controls fetching, not indexing. Some AI agents that fetch a page because a person asked say they may not follow it.
Test yours with the free robots.txt checker and AI crawler checker.