Every AI crawler, and what blocking it actually costs
Most block lists treat every AI user-agent as one thing. They are not. Some fetch your pages so an assistant can cite you; some collect training data. Disallowing the first kind is how sites disappear from ChatGPT without ever meaning to.
Showing 28 of 28 agents. Tick any to build a robots.txt block — the generator warns you when a choice would cost visibility rather than just training data.
AI answer engines
These fetch your pages so an assistant can answer questions about you and link back. Blocking them removes you from that assistant.
Search crawlers
Classic search crawlers. They also build the AI surfaces layered on top of search, so blocking them costs you both.
AI training crawlers
These collect data to train models. Blocking them is a legitimate business decision and costs you no search or assistant visibility.
The distinction that decides everything
In 2023 the question was “should I let AI train on my content?” and the answer was a matter of taste. Sites added a GPTBot block, felt they had taken a position, and moved on.
That block list is now actively harmful, because the agents that decide whether an assistant can citeyou are different agents with different names. A site that blocked “OpenAI” by pattern-matching the operator often blocked OAI-SearchBot and ChatGPT-User alongside GPTBot — and quietly removed itself from ChatGPT.
Training and retrieval are separate decisions with separate consequences. Anyone who tells you to “block the AI bots” without making that distinction is giving advice from two years ago.
Answer engines
Fetch your pages so an assistant can answer and link back. Blocking removes you from that assistant. Rarely the intention.
Search crawlers
Googlebot and bingbot. They also feed the AI surfaces built on search, so blocking them costs both at once. Never correct.
Training crawlers
Collect data for model training. Blocking costs no visibility whatsoever. A legitimate choice in either direction.
A user-agent string is not identification
Anything can claim to be Googlebot, and scrapers routinely do to get past rate limits. If you are making access decisions on user-agent alone, you are trusting a header the client wrote. Verify by reverse DNS with a forward confirmation, or against the published IP ranges — both are linked per agent above where the operator provides them.
Frequently asked questions
Which AI crawlers should I actually block?
Start from the category, not the brand. Blocking a training crawler — GPTBot, ClaudeBot, CCBot, Google-Extended — withholds your content from model training and costs you no search or assistant visibility. That is a legitimate business decision either way. Blocking an answer engine — OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Applebot — removes you from that assistant's results entirely. Almost nobody intends the second thing, and a great many sites have done it by copying a block list.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data to train OpenAI's models. OAI-SearchBot builds the search index ChatGPT uses to find and cite live sources. They are separate agents with separate IP ranges and separate robots.txt tokens. Blocking GPTBot has no effect on whether ChatGPT can find and link your site today; blocking OAI-SearchBot removes you from ChatGPT search. Conflating the two is the most expensive misunderstanding in this entire subject.
Does Google-Extended affect my rankings?
No. Google-Extended is a token, not a crawler — nothing fetches under that name. It controls whether content Googlebot has already collected may be used to train and ground Gemini. Disallowing it has no effect on Google Search rankings or AI Overview eligibility. This is the most persistent myth on the list, and it costs people nothing to disallow it if they want to.
How do I know a request claiming to be Googlebot really is?
User-agent strings are free text and trivially spoofed — scrapers routinely claim to be Googlebot to bypass rate limits. The reliable check is a reverse DNS lookup on the requesting IP, confirming it resolves to googlebot.com or google.com, then a forward lookup confirming that hostname resolves back to the same IP. Google, OpenAI, Anthropic and Perplexity all also publish IP range lists, linked in each entry here where they exist.
Do these crawlers actually obey robots.txt?
The major operators document that they do, and in practice they behave. Two caveats worth knowing. User-triggered agents — ChatGPT-User, Perplexity-User, Claude-User — are treated differently by some operators on the reasoning that a human explicitly asked for the page. And robots.txt is a request, not a wall: it is honoured voluntarily. Anything determined to ignore it will, so if you need enforcement rather than a signal, that has to happen at the firewall or CDN.
Will blocking AI crawlers protect my content from being copied?
Not meaningfully. Blocking the well-behaved agents that identify themselves removes you from AI assistants while doing nothing about scrapers that do not identify themselves at all. Your content also reaches models through Common Crawl, through third parties quoting you, and through anyone who pastes your page into a chat window. If the goal is to be cited accurately rather than to be invisible, the effective strategy is the opposite one — be readable, be structured, and be the source worth quoting.
How often does this list change?
Frequently. New agents appear as products launch, tokens get renamed, and legacy tokens like anthropic-ai linger in robots.txt files for years after the crawler behind them stopped existing. That is precisely why a block list copied from a blog post in 2024 is now both incomplete and partly wrong. Check what your own robots.txt is doing rather than trusting a list — including this one.
Related tools and reading
Most sites blocking AI never decided to.
They pasted a list. Check yours in thirty seconds with the tester, and if the answer surprises you, the rest of the AI visibility picture usually does too.
Get a Free AI Visibility Audit