Is ChatGPT Allowed to Crawl Your Website? How to Check and Control It
Most sites blocking ChatGPT never meant to. Here is how to check which AI crawlers your robots.txt allows, and which ones you actually want to keep out.
A prospect asked us last month why his company never came up when he asked ChatGPT for suppliers in his category. His site ranked fine on Google. His competitors, smaller and newer, were being named.
His robots.txt had four lines in it that blocked every OpenAI user-agent. Someone had pasted them in during a 2024 panic about AI training data, and nobody had looked at the file since. He had spent eighteen months invisible to the assistant his buyers were switching to, and it took ninety seconds to find.
This is common enough that it is now the first thing we check on any new account. Here is how to check yours, and how to decide what you actually want blocked.
The Three Jobs Behind One Label
Almost every argument about AI crawlers goes wrong in the same place: people talk about "AI bots" as one thing. There are three kinds, and they have opposite consequences.
Training crawlers collect pages to train future models. GPTBot, ClaudeBot, CCBot, Bytespider and Google-Extended all sit here. Blocking these costs you nothing in visibility today. Whether you want your writing in someone's training set is a business and licensing question, and reasonable owners land on both sides of it.
Answer crawlers fetch your pages so an assistant can cite you right now. OAI-SearchBot builds the index behind ChatGPT's search. ChatGPT-User fetches a page live when someone asks ChatGPT to look at it. PerplexityBot, Claude-SearchBot, DuckAssistBot and Applebot do the equivalent for their platforms. Block these and you are removed from the answer, permanently, for every question in your category.
Search crawlers are Googlebot and bingbot. They feed classic results and they feed AI surfaces, since AI Overviews are assembled from Google's existing index. There is no version of this where blocking them is correct.
The expensive mistake is treating group one and group two as the same decision. A site that blocks GPTBot has made a defensible call about training. A site that blocks OAI-SearchBot has removed itself from ChatGPT and usually has no idea.
Check What Your Site Actually Allows
Open yoursite.com/robots.txt in a browser. If you see any of these, you are blocking something in the answer group:
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
Reading a robots.txt by eye gets unreliable fast, because the rules that apply to a given bot depend on user-agent matching order, wildcards, and which group a Disallow: / sits in. A file with a catch-all User-agent: * block and several specific groups below it will surprise most people.
Our AI Bot Access Tester resolves it properly. Enter a domain and it tests your live robots.txt against 28 crawler tokens, sorted into the three categories above, and tells you which line is doing the blocking. What we look at first is whether anything in the answer group comes back disallowed, because that is the only group with an immediate revenue cost.
If you want the reference rather than the test, the AI Crawler Directory lists every one of those 28 agents: who runs it, what it collects, whether it respects robots.txt at all, and what blocking it actually costs you.
Not sure whether AI assistants can see your site at all? Get a Free Visibility Audit →
The Configuration We Recommend
For the large majority of service businesses, this is the right shape:
# Search and answer engines: allowed
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Model training: your call
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://yoursite.com/sitemap.xml
Two notes on that file.
The training block is genuinely optional and we do not push clients either way. If your content is your product, blocking makes sense. If you are a plumbing company whose content exists to generate calls, blocking GPTBot buys you nothing and costs nothing. Pick and move on.
The Allow: / lines are not strictly required, since anything not disallowed is allowed by default. They are there so the next person to open the file can see the decision was deliberate. That has saved more accounts than any technical benefit would.
Before you deploy any change, run the file through the robots.txt Tester with a few real URLs and each user-agent you care about. A misplaced trailing slash in a Disallow line can take out a whole directory, and robots.txt has no error reporting to tell you it happened.
Where Blocks Come From When Nobody Added One
Half the blocked sites we find have a robots.txt nobody wrote.
Security plugins. Several WordPress security and SEO plugins added AI-blocking toggles in 2024 and 2025, some enabled by default, some buried in a "privacy" tab. The plugin writes into a virtual robots.txt that does not exist as a file on disk, which is why searching the server for it turns up nothing.
CDN and host defaults. Cloudflare and several other providers now ship AI-crawler blocking as a default or one-click setting on new domains. It works at the network layer, so your robots.txt can say Allow all day while the request is refused before it reaches your server. This is the hardest version to diagnose, because the file looks correct.
Staging configuration promoted to production. A Disallow: / written for a staging site, shipped live during a migration. This blocks Googlebot too, so it usually surfaces within a fortnight as a traffic collapse.
An agency decision from a previous engagement. Worth asking, since it is often documented nowhere.
If the tester reports blocking that your robots.txt does not explain, the block is above the file. Check the CDN bot-management settings before you touch anything else.
What robots.txt Cannot Do
Two limits matter, and both get oversold in the other direction.
Robots.txt is a request, not a control. Well-run crawlers honour it. Others ignore it entirely, and a few undocumented ones do not publish a user-agent at all. If your requirement is legal or contractual rather than preferential, robots.txt is the wrong instrument, and you want authentication or network-level rules.
Allowing a crawler also does not get you cited. It makes citation possible. What gets you named in an answer is the same substance that earns a link: clear statements of fact, prices, service areas, and specifics a model can extract without inference. We covered that mechanism in how to get your business recommended by ChatGPT, and it is the harder half of the job. Crawler access is the easy half, which is exactly why leaving it broken is so wasteful.
The Case for Blocking, Argued Honestly
We recommend allowing answer crawlers to nearly everyone, so it is worth stating when the opposite is right.
Blocking makes sense if your content is licensed or sold, if you publish original research that competitors would otherwise get summarised for free, or if you are a publisher negotiating a content deal and access is your leverage. Some news organisations have taken exactly this position and been right to.
It does not make sense as a defensive reflex. The most common reasoning we hear is that AI answers cannibalise clicks, so denying the crawler protects traffic. The mechanism does not work that way. Blocking removes you from the answer; it does not remove the answer. The assistant recommends three competitors instead, and the buyer never learns you exist.
For a local service business choosing between a summarised mention with your name in it and no mention at all, that is not a close call.
Check It Quarterly, Not Once
Crawler tokens change. OpenAI split its single GPTBot agent into three distinct user-agents as the product line grew, and other operators have done the same. A robots.txt written against the 2024 list is already out of date, and the blocks it contains keep applying to agents whose behaviour has since changed.
Put a recurring reminder against it, run the access tester each quarter, and read the diff. It takes two minutes and it is the cheapest recurring check in AIEO.
For the primary sources, OpenAI publishes its current agents at platform.openai.com/docs/bots, and Google maintains its crawler list in Search Central. Both change without announcement.
The Short Version
Find out what you are blocking before you decide what you want to block. Most sites are not making the trade they think they are: they set out to keep their writing out of a training set and quietly removed themselves from the assistant their buyers now ask first.
Training crawlers are a licensing question with no visibility cost either way. Answer crawlers are a visibility question with no licensing benefit to blocking. Once you separate the two, the configuration takes ten minutes and rarely needs revisiting.
Getting good Google rankings while AI assistants recommend your competitors?
A Free Visibility Audit covers your crawler access across all 28 AI and search agents, and shows which of your pages an assistant can actually extract an answer from.
Related reading: What Is AIEO | How to Get Your Business Recommended by ChatGPT | What Is Google AI Overview
Search Beyond Google
Search Beyond Google is a digital marketing growth agency helping ambitious businesses in the GTA and across North America build compounding visibility across SEO, Local SEO, AEO, AIEO, Google Ads, and Social Media. Every article is researched and written by the SBG team — practitioners who build and test these strategies daily across real client campaigns.
Ready to apply this to your business?
A free 45-minute Visibility Audit maps these frameworks to your specific market, competitors, and current digital presence.
Get Your Free Visibility Audit →Related Articles
Why AI Crawlers Do Not Run Your JavaScript (and What It Costs You)
Googlebot renders JavaScript. Most AI crawlers do not. If your content only exists after a script runs, assistants see an empty page while Google ranks you fine.
How to Rank on Google Gemini and Microsoft Copilot
Gemini and Copilot pull from different indexes and reward different signals. Here's how each one picks sources and what to change to get your business cited.
How to Get Your Business Featured on Perplexity AI
Perplexity AI is growing 15% month-over-month and is the go-to research tool for professional B2B buyers. Here is exactly how to get your business cited in Perplexity responses.