Two lines of text can make your business invisible to ChatGPT.
User-agent: *
Disallow: /
Somebody added them to keep a test site out of Google. The site went live. The lines stayed. Nobody noticed, because the site still loads fine for people. It just doesn't load for the crawlers that feed AI search, and AI can't name what it was told not to read.
This free check reads your robots.txt and a few of your pages the way a polite crawler does, and tells you, crawler by crawler, who may come in.
Not all AI crawlers do the same job
This is the part most robots.txt advice gets wrong. There are three kinds of AI bots, and blocking each one costs you something different.
Search crawlers decide whether you can appear in AI answers. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Perplexity says PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is not used to train its models. Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." And Google says a page must be indexed in Google Search, which means Googlebot must be able to crawl it, to be shown as a supporting link in AI Overviews or AI Mode. Block these, and you've asked to be left out.
Training crawlers feed future models. GPTBot collects content that "may be used in training" OpenAI's models. ClaudeBot does the same for Anthropic. Blocking them is a choice about training, and plenty of sites make it on purpose. One needs care: Google says its Google-Extended token covers training future Gemini models and grounding in the Gemini apps, while it "does not impact a site's inclusion in Google Search."
User-triggered fetchers read a page because a person asked. When someone asks ChatGPT or Perplexity about a specific page, ChatGPT-User or Perplexity-User may fetch it. OpenAI says robots.txt rules "may not apply" to these, and Perplexity says its fetcher "generally ignores robots.txt rules," because a human asked.
So "block all AI" and "allow all AI" are both blunt answers to a question with three parts.
What the check reads, line by line
Give it a website. It reads your robots.txt and up to three pages, politely, as ScanFreeBot, and calls no AI at all. Then it reports:
The search crawlers, one line each: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Googlebot and Bingbot. Each line says whether your robots.txt lets that crawler read your home page, and shows the exact lines of your file that decided it, so you can find them and fix them. If a search crawler can read your home page but not your key pages, that gets its own line.
The training crawlers and user fetchers, GPTBot, ClaudeBot, Google-Extended, CCBot, ChatGPT-User and Perplexity-User, reported as information, not failures, because blocking them is often deliberate.
Your firewall. Whether your site answers an honest, identified crawler at all, or turns it away with an error or a challenge page. If it turns ours away, it may be turning AI crawlers away too.
Indexing basics: whether robots.txt was found, whether your home page says "noindex" in a tag or a header, its canonical address, whether your key pages answer, and whether your home page has real text before JavaScript runs. Many AI crawlers don't run JavaScript; an empty shell reads as an empty page.
Facts AI can read: whether one phone number appears on every page we read, and whether your contact details sit where visitors look for them.
Every line comes with the evidence behind it: the robots.txt lines, the HTTP status, the tag. No grades. Pass, fail, partial, or "check this by hand" when we couldn't tell.
What it doesn't do
It doesn't grade llms.txt. It tells you whether the file is there, for your information only, and never recommends adding one: Google says no AI text files are needed for its AI features, and our scans mark the file informational only.
It can't see your site from the AI companies' own addresses. It tests our identified crawler; a firewall that treats OpenAI's or Perplexity's published IP ranges differently would need checking in its own settings. If your site sits behind Cloudflare, look there first: Cloudflare changed its default in July 2025 to block AI crawlers, and Perplexity publishes instructions for allowing its bots through a firewall.
And it doesn't tell you whether AI names you. Being allowed in is the door, not the conversation. To see what the AIs actually say, run a free scan.
If the door is shut, nothing else you do will be read. Check the door first.
Questions
Can AI crawl my website?
Probably, unless your robots.txt or your firewall says no. The AI companies publish their crawlers' names and say they follow robots.txt, so the file is the first place to look. This check reads it and tells you, crawler by crawler, which ones may read your home page and your key pages.
How can I check if my website is crawlable?
Read your robots.txt (it lives at yoursite.com/robots.txt), look for "noindex" tags on your pages, and make sure your text is in the page before JavaScript runs. Or run this check, which does all three and shows the exact lines behind each result.
Is ChatGPT a web crawler?
ChatGPT itself isn't, but OpenAI runs crawlers for it. OAI-SearchBot finds pages for ChatGPT's search answers. GPTBot collects content that may be used for training. ChatGPT-User fetches a page when a person asks about it. Each one can be allowed or blocked separately in robots.txt.
How to protect your website from AI crawlers?
Decide what you're protecting against. To keep your content out of model training, block the training crawlers, such as GPTBot and ClaudeBot, in robots.txt. That doesn't remove you from AI search. Blocking the search crawlers, such as OAI-SearchBot or PerplexityBot, does, so do it only if you mean it.
Should I block AI crawlers on Cloudflare?
Only the ones you mean to. Cloudflare changed its default in July 2025 to block AI crawlers, so a site behind it may be blocking the search crawlers that put you in AI answers without anyone choosing that. Check your Cloudflare bot settings, allow the search crawlers you want, and block training crawlers if that's your choice.
How can I stop AI from scraping my website?
Robots.txt asks crawlers to stay out, and the major AI companies say their crawlers honour it, but it's a request, not a lock: the standard says its rules "are not a form of access authorization." Fetches triggered by a user's question may ignore it. For harder limits, use your firewall or CDN's bot controls, knowing that blocking AI search crawlers also keeps you out of AI answers.
Sources
| Claim | Source | Checked on |
|---|---|---|
| OAI-SearchBot: sites opted out "will not be shown in ChatGPT search answers"; GPTBot: content "may be used in training"; ChatGPT-User: "robots.txt rules may not apply"; each setting independent | OpenAI: Overview of OpenAI crawlers | 15 Nov 2026 |
| PerplexityBot: "designed to surface and link websites in search results on Perplexity", not used for training; Perplexity-User "generally ignores robots.txt rules"; firewall instructions | Perplexity: Perplexity crawlers | 15 Nov 2026 |
| Claude-SearchBot: blocking "may reduce your site's visibility and accuracy in user search results"; ClaudeBot: training; Claude-User: user-initiated; Anthropic honours robots.txt | Anthropic: Does Anthropic crawl data from the web? | 15 Nov 2026 |
| Google-Extended: controls use for training future Gemini models and for grounding in Gemini Apps and Vertex AI; "does not impact a site's inclusion in Google Search" (page last updated 30 Oct 2026) | Google: Google's common crawlers | 15 Nov 2026 |
| AI Overviews / AI Mode: a page must be indexed and snippet-eligible in Google Search; no AI text files needed | Google Search Central: AI features and your website | 15 Nov 2026 |
| Cloudflare changed its default to block AI crawlers (1 Jul 2025) | Cloudflare blog: Content Independence Day (1 Jul 2025) | 10 Nov 2026 |
| robots.txt rules "are not a form of access authorization" (RFC 9309) | RFC 9309: Robots Exclusion Protocol | 10 Nov 2026 |