Findable & Crawlable

What it measures (out of 25)

Before any other optimization matters, AI search engines need to be able to discover, reach, and read your site. AI crawlers are distinct from the traditional web crawlers you may be familiar with — each major platform (ChatGPT, Perplexity, Claude, Google AI Overviews) runs its own crawler under its own user-agent string, and those crawlers are not automatically covered by the same permissions you grant to Googlebot. This category checks both the access layer (whether the right crawlers are allowed in) and the foundational discovery hygiene (HTTPS, sitemap, canonical signals) that lets engines find and trust your pages in the first place. Failing the access checks doesn't just lower your score — it means the affected platform cannot index your content at all, regardless of how well-optimized everything else is.

What we check for

• ›Whether each major AI crawler is explicitly allowed in your robots.txt, including platform-specific bots that do not inherit permissions from general allow-all rules
• ›Whether your site delivers rendered HTML to crawlers rather than an empty JavaScript shell that AI bots cannot parse
• ›Whether any CDN-level security rules — including managed firewall settings that block AI crawlers by default — are inadvertently blocking AI bot access
• ›Whether foundational discovery hygiene is in place: HTTPS, a valid sitemap, clean canonical tags, and proactive indexing signals

Example finding

A home-services site had a clean robots.txt for traditional search crawlers but had activated a CDN security rule that blocked all AI user-agents by default — resulting in a near-zero score on the platform-access checks and no presence in any AI-generated local recommendations.

Why this matters

AI search engines maintain separate crawler fleets that use different user-agent identifiers than their web-search counterparts. A robots.txt that says "allow all" for Googlebot does not automatically allow the AI-specific crawlers, and a security rule intended to reduce bot traffic can silently exclude every AI platform at once. Passing this category does not guarantee citation — crawl access is a prerequisite, not a ranking signal — but failing it makes citation impossible on the blocked platforms. Getting the findability layer right is the foundational first step.