Why analytics scripts can't see AI crawlers
Analytics tools like Google Analytics count a visit when their script runs in a browser. AI crawlers and assistants request your HTML from their own servers and never run that script, so they never appear. Google Analytics also excludes known bots and spiders automatically, and you can't turn that off. The only record of an AI crawler's visit is on your server, your CDN or your hosting platform.
Step 1: get the requests
You need, for each request: the time, method, host, path, status code, user agent and client IP address.
- Your web server. Nginx and Apache access logs have all of these by default.
- A CDN. Cloudflare, CloudFront and Fastly can export request logs; a Worker or edge function can also record them as they happen.
- A hosting platform. Vercel and Netlify offer log drains on their paid tiers (Vercel Pro and Enterprise, Netlify Enterprise).
If you sit behind a CDN or load balancer, make sure the IP in your logs is the visitor's, not the proxy's.
Step 2: keep the requests that matter
Keep requests for pages (HTML, GET or HEAD) and for robots.txt, llms.txt and sitemaps. Drop images, scripts and styles. Then keep only clients that aren't mainstream browsers. What's left is crawlers, AI fetchers, scripts and headless browsers.
Step 3: name each agent
Match the user agent against the agents AI companies document, as whole product tokens: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Claude-SearchBot and the rest in our AI crawler user agents table. Group them by kind: fetches for a person, AI search, and training.
Step 4: verify the address
This is the step most reports skip. A user agent is only a claim. Download each operator's published IP list (for example OpenAI's), refresh it daily, and check whether each request's address falls inside it:
- Verified: the address is in the operator's list.
- Claimed: the operator publishes no list, so you can't tell.
- Impostor: the operator publishes a list and the address isn't in it.
Impostors are common: scrapers borrow well-known crawler names to get past filters. Counting them as GPTBot overstates how much OpenAI reads your site.
Step 5: turn it into decisions
Once the requests are named and verified, a few reports answer most questions:
| Report | What it tells you |
|---|---|
| Fetches for a person, by page | Which pages people ask AI assistants about |
| Pages AI reads but nobody arrives at from AI | Content that informs answers without earning clicks |
| Errors served to AI agents | Pages that are broken or blocked for assistants |
| robots.txt and llms.txt fetches | Whether crawlers read your rules |
| Impostors, by claimed agent | How much "AI traffic" isn't |
Do it without building it
LogZuno's AI visibility does all five steps. Point a collector or log drain at it, and every request is classified, verified against the operators' lists (refreshed daily), and reported, with IP addresses used once and never stored.