About AI Crawl Check

What it is

AI Crawl Check is a free, no-signup tool that shows how a public website treats AI crawlers. It is an independent project and is not affiliated with OpenAI, Anthropic, Perplexity, Google, Cloudflare, or any other crawler operator.

What a check does

When you enter a URL, the tool uses only its origin (scheme and host) and makes these public requests from its server:

• /robots.txt, parsed for 14 AI user-agent tokens (for example GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended). It also looks for Cloudflare's managed robots.txt markers.
• /llms.txt and /llms-full.txt, reporting status, content type, size, and basic structure.
• The homepage (/), fetched twice: once with a browser-style user agent and once with a GPTBot user agent. A different response can point to an edge or firewall rule that robots.txt does not show.

Requests time out after 9 seconds, follow only a few redirects, and are refused for hosts that resolve to private or local network addresses.

Limits of the results

• Results are a single point-in-time observation of public HTTP responses.
• robots.txt states a preference. OpenAI says robots.txt rules may not apply to the user-triggered ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, so those two are shown as “Disallowed” rather than “Blocked”.
• Google-Extended has no separate user agent, so it can only be checked in robots.txt, not live.
• The live probe sends a GPTBot user-agent name from this tool's server, not from OpenAI's published IP ranges. Services that verify crawlers by IP address may treat the real GPTBot differently.
• An Allow rule does not guarantee indexing, citation, or training use.

Sources

Crawler behavior notes are based on the operators' own documentation: OpenAI, Perplexity, Anthropic, Google, and Cloudflare. robots.txt parsing follows RFC 9309.

See the privacy page for what is and isn't stored.