Free tool
robots.txt checker for AI crawlers.
Which of 12 AI and search crawlers your robots.txt lets in, on your home page and key paths, with the lines that decided each one. One request, no sign-in.
Type a page address to check that path too. 10 runs an hour per visitor. Only public websites.
How it reads the file. The way the crawlers do.
- The crawlers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Googlebot, Bingbot, GPTBot, ClaudeBot, Google-Extended, CCBot, ChatGPT-User, Perplexity-User.
- The paths: your home page, the page you typed, and /about, /contact, /services, /products, /blog, /pricing.
- A crawler named in its own group follows only that group; the others follow the group for every crawler (User-agent: *). The longest matching rule wins, and Allow wins a tie.
- No robots.txt at all means every crawler may read every page.
- robots.txt is a request. It doesn't show whether a firewall turns a crawler away: the AI crawler checker tests that.
For the whole picture (the firewall, noindex, canonical and text that needs JavaScript), use the AI crawler checker.
Then see what AI says about you.
The free scan asks the AI apps your customers' questions and shows who they named.