Free tool · No sign-up
AI crawler checker: which bots can read your site?
Most site owners have never checked, and a good number have blocked the wrong ones. This reads your robots.txt and tells you not just what is allowed, but what each decision actually costs you.
What it checks
The 16 crawlers this checks
Each one is matched against your robots.txt using the same rules the crawler itself follows: the most specific user-agent group wins, then the longest matching path.
| Crawler | Run by | Used for | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | AI answers | Blocking removes you from ChatGPT search results. This is the one that decides whether ChatGPT can cite you. |
ChatGPT-User | OpenAI | AI answers | Fetches a page when a ChatGPT user follows a link or asks about your site. Blocking means those users get nothing. |
GPTBot | OpenAI | Model training | Training data only. Blocking does NOT remove you from ChatGPT search — that is OAI-SearchBot. |
Claude-SearchBot | Anthropic | AI answers | Indexes pages so Claude can cite them in answers. Blocking removes you from those citations. |
Claude-User | Anthropic | AI answers | Fetches a page on behalf of a user asking about it. Blocking means Claude cannot read your page for them. |
ClaudeBot | Anthropic | Model training | Training data only. Blocking does not affect whether Claude can cite you in answers. |
PerplexityBot | Perplexity | AI answers | Builds the index Perplexity answers from. Blocking removes you from its results. |
Perplexity-User | Perplexity | AI answers | User-initiated fetch. Blocking affects people actively asking about your site. |
Google-Extended | Model training | Gemini training and grounding ONLY. Blocking has no effect whatsoever on your Google Search ranking. | |
Googlebot | Search | Classic Google Search, and the crawler behind AI Overviews. Blocking this removes you from Google entirely. | |
Bingbot | Microsoft | Search | Bing search and Microsoft Copilot. Blocking removes you from both. |
Applebot-Extended | Apple | Model training | Apple Intelligence training. Blocking does not affect Siri or Spotlight results, which use Applebot. |
Meta-ExternalAgent | Meta | Model training | Meta AI training. No effect on Facebook or Instagram link previews. |
CCBot | Common Crawl | Model training | Common Crawl is a public dataset many models train on. Blocking it reaches a wide set of downstream users at once. |
Bytespider | ByteDance | Model training | TikTok/ByteDance model training. Widely reported to crawl aggressively. |
Amazonbot | Amazon | Model training | Alexa answers and Amazon model training. |
The distinction that matters
Training crawlers and answer crawlers are not the same thing
Blocking GPTBot keeps your content out of OpenAI's training data. It does not stop ChatGPT citing you — that is OAI-SearchBot. Plenty of sites have blocked the first believing they did the second, and quietly removed themselves from nothing at all. Others blocked the answer crawler and wondered why they vanished from AI results.
The same confusion surrounds Google-Extended, which governs Gemini training and grounding and has no bearing whatsoever on your Google Search ranking. Googlebot is the one that matters there, and it is also the crawler behind AI Overviews.
Most tools in this space print a list of user agents and a tick or a cross. The column worth reading is the one that says what happens next.
robots.txt example
Stay in AI answers, stay out of AI training
This is the setup most businesses actually want. It lets the crawlers that cite you in ChatGPT, Claude and Perplexity answers read your site, keeps the training crawlers out, and leaves Google and Bing search untouched. Put it in the robots.txt file at the root of your domain, then run the checker above to confirm it reads the way you meant.
# Stay in AI answers and search results
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Keep content out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /On Wix, open SEO in your dashboard and use the robots.txt editor under Tools and settings. Changes apply the next time each crawler fetches the file; Google rechecks it at least once a day.
While we are here
On llms.txt
There is a lot of advice telling you to add an llms.txt file. The measured
evidence does not support it: across more than 500 million AI bot visits in a 90-day
window, 408 fetched the file. Google's 2026 guidance says plainly it is
not needed for AI Overviews or AI Mode, and no major model provider has committed to
reading it.
It does do real work as a routing map for AI coding agents — Cursor, Claude Code, Copilot and similar. That is a decent reason to have one if you publish developer documentation. It is not a reason to expect more AI search visibility, and we would rather say so than sell you a file.
AI crawlers · Questions
What people get wrong about this
What does this tool check?
It reads your robots.txt and works out whether each major AI crawler is allowed to fetch your site, using the same grouping and longest-match rules crawlers use. It then explains what blocking each one actually does, which is the part most owners get wrong.
Does blocking GPTBot stop ChatGPT from citing my site?
No, and this is the single most common mistake. GPTBot collects training data. The crawler that decides whether ChatGPT can cite you in search results is OAI-SearchBot, and ChatGPT-User is the one that fetches a page when someone asks about it directly. Blocking GPTBot protects your content from training while leaving you visible in answers — which is what most businesses actually want.
Does blocking Google-Extended hurt my Google ranking?
No. Google-Extended controls Gemini training and grounding only. It has no effect on Google Search ranking at all. Googlebot is the crawler behind both classic Search and AI Overviews, so blocking that one would remove you from Google entirely.
Should I create an llms.txt file?
Probably not for AI search. Across a 90-day window of more than 500 million AI bot visits, only 408 fetched llms.txt — GPTBot, ClaudeBot, PerplexityBot and Google-Extended overwhelmingly skip it and crawl HTML directly. Google stated in 2026 that it is not needed for AI Overviews or AI Mode, and no major model provider has committed to using it. It does have a genuine use as a routing map for AI coding agents such as Cursor, Claude Code and Copilot. Just do not expect it to change your visibility in AI answers.
I have no robots.txt. Is that bad?
Not in itself. No robots.txt means nothing is disallowed, so every crawler is allowed. That is a permissive default rather than a broken one. It only becomes a problem if you intended to restrict something.
Should I block AI training crawlers?
It depends on what your content is worth to you and it is a business decision rather than a technical default. Publishers with original research often block training while allowing answer crawlers. A services business usually wants maximum visibility and blocks nothing. What matters is that the decision is deliberate — most sites have simply never looked.
Want this set deliberately?
Which AI systems may read your content, and which may train on it, is a business decision. Most sites have simply never made it. We will audit what you are allowing today and set it to match what you actually want.