Definition
Crawler directives are instructions website owners give crawlers. They come in three places:
| Where | Controls | Example |
|---|---|---|
| robots.txt | Which URLs a crawler may fetch | Disallow: /admin |
| Meta robots tag | Whether a page may be indexed and its links followed | <meta name="robots" content="noindex"> |
| X-Robots-Tag header | The same, for any file type | X-Robots-Tag: noindex |
Fetching vs indexing
robots.txt controls fetching; meta tags and headers control indexing. A page blocked in robots.txt can't be fetched, so a noindex on it is never seen.
AI crawlers
AI companies document how their crawlers treat robots.txt. Some agents that fetch a page because a person asked may not follow it; see AI crawler user agents.