Spot AI crawlers in your server logs

WebAISEO: free AI-search check All guides WebAISEO Terms For AI agents

Published 12 November 2026 · by TrueTest

Your web server's access log records every request, including the ones from AI crawlers, so it is the place to see whether they visit, which pages they read and what your server answers them. What our check cannot do: it reads one page from the outside, so it cannot see your logs, count crawler visits or tell a genuine crawler from one that only borrows its name. Those answers are in your own logs, and this page shows how to read them.

Check what my robots.txt allows See a full example first

Type your address into the box at the top of the WebAISEO page: the Readiness basics check reads one public page and its robots.txt, sitemap and llms.txt, takes a few seconds, needs no sign-up and is free. The paid audits are described on the WebAISEO page.

Which names should I search for?

Each AI company puts a name in the user-agent string of its requests. Search your log for these: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity), and bingbot. Google-Extended never appears: Google says it has no user agent of its own, and Googlebot does the fetching.

What do the lines look like?

SAMPLE lines, made up for this page on 12 November 2026 in the common "combined" log format; not our own traffic, and the address is left out:

[address] - - [12/Nov/2026:06:14:02 +0000] "GET /pricing HTTP/1.1" 200 18342 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
[address] - - [12/Nov/2026:06:20:47 +0000] "GET /robots.txt HTTP/1.1" 200 214 "-" "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
[address] - - [12/Nov/2026:06:31:09 +0000] "GET /contact HTTP/1.1" 403 162 "-" "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"

The number after the request is the status: 200 means the page was served, 403 means your server or firewall refused it. A crawler that reads robots.txt and then gets 403 on every page is the sign of a block outside robots.txt (for example at Cloudflare).

How do I know a crawler is genuine?

Anyone can copy a crawler's name into their own requests. OpenAI, Perplexity and Anthropic publish the address ranges their crawlers use, as JSON files linked from their documentation. Compare the address of a request with the list for the crawler it claims to be; a request whose address is not on the list is not from that company, whatever its name says.

Do all of them obey robots.txt?

Not in the same way. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person asked for the page, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says its bots honour robots.txt directives. Check each company's own page (listed under Sources) before you rely on robots.txt alone.

What do the counts tell me?

That a crawler can reach and read your pages, which is the first step. They do not tell you that an assistant cites you: a page can be read every day and never quoted. Count which pages are fetched, and whether those are the pages you want quoted.

Sources (read 12 November 2026)

Outside companies change their documentation; we review these sources by 12 February 2027.

Questions people ask

Where is my access log?

It depends on your host: control panels usually offer it under Logs or Statistics; on your own server it is the web server's access log file.

Should I block crawlers that ignore robots.txt?

That is your choice. A firewall rule enforces it; remember that user-started fetchers bring a person's question to your page.

Why do I see a crawler but no visitors from AI answers?

Reading a page and citing it are different steps; and visitors from AI answers often arrive without a referrer.

Is the check free?

Yes, for one page with no sign-up.