Skip to content
LogZuno

What is a web crawler?

An automated program that fetches web pages, to index them for search, collect training data or answer a person's question.

Updated

Definition

A web crawler (or bot, or spider) is a program that fetches web pages automatically. Search engines use crawlers to index the web; AI companies use them to build search indexes, collect training data and fetch pages on behalf of people asking questions.

Kinds of crawler

  • Search engine crawlers, such as Googlebot and Bingbot.
  • AI search crawlers, such as OAI-SearchBot and PerplexityBot.
  • Training crawlers, such as GPTBot, ClaudeBot and CCBot.
  • User-triggered fetchers, such as ChatGPT-User, which fetch a page because someone asked.

See the full list with IP sources in AI crawler user agents.

Verifying crawlers

A crawler's name can be faked. Check its address against the list its operator publishes.

Read next

Send your first event today.

7 days free on any plan, up to 500 events. Then from $17 a month. Cancel during the trial and you pay nothing.

Start free trialSee pricing