Sitemaps, HTTPS and speed for AI crawlers

WebAISEO: free AI-search check Prices WebAISEO Terms For AI agents

Published 11 November 2026 · by TrueTest

Before an engine can read a page it has to find it and fetch it without trouble. The discoverability area (15% of the score) checks the basics that make a site easy to crawl.

Check my site's basics free See a full example first

Type your address into the box at the top of the WebAISEO page: the Readiness basics check reads one public page and its robots.txt, sitemap and llms.txt, takes a few seconds, needs no sign-up and is free. The paid audits are described on the WebAISEO page.

What does the check test?

Check	Pass	Points
HTTPS	the page's final address starts with https://	5
XML sitemap	the sitemap named in robots.txt (or /sitemap.xml) answers with XML	6
llms.txt	/llms.txt answers with text (optional, labelled emerging)	2
Fast response	the page answers in under 1.5 seconds (warn under 4 s)	2

How do I name the sitemap in robots.txt?

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Use the full address. Engines also accept a sitemap submitted in their webmaster tools, but the robots.txt line is read by every crawler that reads robots.txt.

What should the sitemap hold?

Every public page you want found, with its canonical address (the same one as its <link rel="canonical">).
No private, redirected or noindex pages.
A lastmod date that changes only when the page changes; ours changes only when a page's content hash changes.

Why does speed matter to a crawler?

Crawlers fetch many pages on a time budget. A page that takes several seconds to answer may be fetched less often, or not at all. The check times how long our server takes to fetch the page's HTML, not how long the page takes to draw in a browser.

Can I tell engines a page changed?

Yes: IndexNow lets a site tell Bing and several other engines about changed addresses at once. We ping it after every change to our own pages.

More guides: Schema and structured data for AI search · GEO and AEO audit: what it checks · Can AI crawlers read my JavaScript site? · Do noindex or nosnippet block AI answers? · AI readiness check as an MCP tool · Headings that AI answers can lift · Author, dates and trust signals · AI search optimisation vs classic SEO

Questions people ask

Is a sitemap required?

No, but without one engines rely on links to find pages; the check gives a warn, not a fail.

My site redirects http to https. Does that pass?

Yes: the check follows up to 4 redirects and judges the final address.

Does a slow check mean my site is slow?

It is one timing of one page from our server; run it twice if a result looks odd.

Is the check free?

Yes, with no sign-up.