Is Cloudflare blocking AI crawlers?

WebAISEO: free AI-search check All guides WebAISEO Terms For AI agents

Published 12 November 2026 · by TrueTest

If your site runs behind Cloudflare, AI crawlers can be blocked before they reach your server, even when robots.txt welcomes them. What our check cannot see: it fetches your page with its own name (it introduces itself as a TrueTest readiness check), not as GPTBot or ClaudeBot, so a block that Cloudflare applies to known AI crawlers may not show up in our result. Our check reads what your robots.txt says and what your server answers to us; to see what Cloudflare does to the AI crawlers, look at your Cloudflare settings and logs.

Check what my robots.txt says See a full example first

Type your address into the box at the top of the WebAISEO page: the Readiness basics check reads one public page and its robots.txt, sitemap and llms.txt, takes a few seconds, needs no sign-up and is free. The paid audits are described on the WebAISEO page.

What does Cloudflare's setting do?

Cloudflare's documentation (updated 2 November 2026) describes AI Crawler and Agent access policies, earlier called AI bot policies. They sort AI traffic by what it is for (Search, Agent and Training) and set an action for each: Block across the zone, Block on pages with ads, Allow, or Disallow for training, which asks crawlers through robots.txt not to use the content. Each policy covers the bots Cloudflare recognises for that purpose and matching unrecognised ones. Cloudflare's own page also points out that robots.txt directives are voluntary and crawlers can ignore them; a Block at the edge is enforced.

Where do I find it?

In the Cloudflare dashboard, according to the same page: Security, then Settings, filter by Bot traffic, then AI Crawler and Agent access policies. Check the Search row in particular: blocking AI search crawlers keeps you out of those engines' answers, while blocking training is a choice many sites make on purpose.

robots.txt says yes, the server says 403: what is happening?

robots.txt is a request to crawlers, read from a file. A firewall or CDN rule is enforced on each request. When the two disagree, the crawler reads "you may come in" and then gets a 403 (forbidden) or a challenge page. In your Cloudflare security events or server logs this shows as requests from GPTBot, ClaudeBot or PerplexityBot answered with 403. Our check can only see it when the rule also blocks our own request, for example a block on every unknown bot.

What should I check?

Run the free check: if robots.txt already blocks a search crawler, fix that first.
Open the Cloudflare setting above and compare each row with what you want.
Look for 403 answers to AI crawlers in your logs after any change.

Sources (read 12 November 2026)

Cloudflare docs: AI Crawler and Agent access policies

Outside companies change their documentation; we review these sources by 12 February 2027.

Questions people ask

Is blocking AI training crawlers bad for me?

It is a legitimate choice. Our check counts a blocked training crawler as a warn, and a blocked search crawler as a fail.

Why does your check pass when AI crawlers are blocked?

Our request carries our own name, so a block on known AI crawlers does not apply to it. Check the Cloudflare setting and your logs.

Do other CDNs and hosts do this too?

Some hosts and firewall plug-ins offer similar switches. The same check applies: compare robots.txt with what the server answers.

Is the check free?

Yes, for one page with no sign-up.