AI crawler robots.txt checker

WebAISEO: free AI-search check Prices WebAISEO Terms For AI agents

Published 11 November 2026 · by TrueTest

Your robots.txt may be blocking AI crawlers without you knowing: some CMS plug-ins, hosting settings and CDN options add blocks for AI bots, and an old User-agent: * / Disallow: / blocks everyone. The free check reads your robots.txt and tells you, crawler by crawler, which ones can read your page.

Check my robots.txt free See a full example first

Type your address into the box at the top of the WebAISEO page: the Readiness basics check reads one public page and its robots.txt, sitemap and llms.txt, takes a few seconds, needs no sign-up and is free. The paid audits are described on the WebAISEO page.

Which AI crawlers does the check test?

Crawler (user agent)	Company	What it is for
`GPTBot`	OpenAI	training
`OAI-SearchBot`	OpenAI	ChatGPT search
`ChatGPT-User`	OpenAI	ChatGPT fetching a page for a user
`ClaudeBot`	Anthropic	training
`Claude-SearchBot`	Anthropic	Claude search
`PerplexityBot`	Perplexity	search answers
`Google-Extended`	Google	use in Gemini / AI features (a robots.txt token, not a separate crawler)
`Bingbot`	Microsoft	Bing and Copilot
`Applebot-Extended`	Apple	use in Apple Intelligence (a robots.txt token)
`CCBot`	Common Crawl	an open crawl that many models are trained on

Search crawlers or training crawlers: what is the difference?

Search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Bingbot) fetch pages to answer questions and link to sources. Training crawlers collect text to train models. Our check treats them differently:

a blocked search crawler is a fail (12 points): you cannot be cited by that engine;
a blocked training crawler is only a warn (4 points): blocking training is a legitimate choice, though it can reduce how well models know your brand.

Example rules

Allow AI search, block training:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Rules are read per crawler: a crawler follows the group with its own name, and only uses * when it has none.

What else can block AI crawlers?

A noindex, nosnippet, noai or noimageai meta tag or X-Robots-Tag header: the check looks for all four.
A firewall or CDN "block AI bots" switch: robots.txt then says yes but the server says no. The check sees what your server answers to a named check, not every crawler's network.

More guides: Schema and structured data for AI search · GEO and AEO audit: what it checks · Do noindex or nosnippet block AI answers? · AI readiness check as an MCP tool · Headings that AI answers can lift · Author, dates and trust signals · Sitemaps, HTTPS and speed for AI crawlers · AI search optimisation vs classic SEO

Questions people ask

Should I block GPTBot?

It is your choice. Blocking GPTBot stops OpenAI training on your pages; it does not stop ChatGPT search, which uses OAI-SearchBot and ChatGPT-User.

Is Google-Extended a crawler?

No. It is a robots.txt token that controls use in Gemini and Google's AI features; Googlebot does the crawling.

My robots.txt is missing. Is that bad?

Without one, every crawler may read everything, so nothing is blocked. The check still suggests adding one that names your sitemap.

Does the check read my whole site?

No. The free check reads one page and its robots.txt, sitemap and llms.txt.