AI crawler checker: which AI bots can access your site?
See how your site treats GPTBot, ClaudeBot, PerplexityBot and 11 more AI bots: robots.txt rules, Cloudflare managed rules, llms.txt, and live edge behavior in one fast scan.
We only make public requests. Your URL is not stored. Privacy
A clearer signal
Four checks. One useful answer.
Read robots.txt
We resolve the rules for citation and training crawlers separately.
Spot Cloudflare
Managed injection markers and AI disallow blocks get their own warning.
Inspect llms.txt
We check availability, content type, and the structure of both files.
Probe the edge
A browser/GPTBot mismatch can reveal a block robots.txt misses.
FAQ
Does this change my robots.txt?+
No. AI Crawl Check only fetches public endpoints and reports what it sees.
What does Mixed mean?+
Mixed means robots.txt blocks some of the checked crawlers but not all, at least one crawler is only partially allowed or has no applicable rule group, or the live GPTBot probe got a different response than a normal browser.
Does a robots Allow guarantee citation?+
No. Robots rules are one input. Provider policies, WAF rules, authentication, and content quality also matter.
Why are ChatGPT-User and Perplexity-User shown as Disallowed instead of Blocked?+
They are user-triggered fetchers. OpenAI says that for ChatGPT-User robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores robots.txt rules. A Disallow rule states your preference, but it may not stop these requests.
Why can't Google-Extended be live-tested?+
Google-Extended has no separate user agent. Google crawls with its existing user agents and uses the Google-Extended token only in robots.txt, so AI Crawl Check reports its robots.txt rule only.