What is the X-Robots-Tag?

An HTTP response header that gives crawlers indexing instructions for any file, including PDFs and images.

Updated 23 October 2026

Definition

The X-Robots-Tag is an HTTP response header that carries the same instructions as the meta robots tag, but works for any type of file, not just HTML pages.

X-Robots-Tag: noindex, nofollow

When to use it

Files that can't carry a meta tag: PDFs, images, feeds.
Applying a rule to many URLs at once from server configuration.

Like the meta tag

The crawler has to be allowed to fetch the file to see the header. Don't also block it in robots.txt.

GlossaryWhat is the meta robots tag?The meta robots tag tells search engines whether to index a page and follow its links. Its common values, and why a blocked page never sees it.Read
GlossaryWhat is a noindex tag?Noindex tag: a directive that tells search engines not to show a page in their results. A plain-language definition from the LogZuno glossary.Read
GlossaryWhat are crawler directives?What are crawler directives? Instructions that tell crawlers what they may fetch, index or follow, given in robots.txt, meta tags or HTTP headers.Read

Send your first event today.

7 days free on any plan, up to 500 events. Then from $17 a month. Cancel during the trial and you pay nothing.

Start free trial See pricing

What is the X-Robots-Tag?

Definition

When to use it

Like the meta tag

Read next

Send your first event today.