Can ChatGPT, Claude and Perplexity actually read your site?

No, and this is the single most common misunderstanding in AI SEO. GPTBot collects training data. A completely different crawler, OAI-SearchBot, builds the search index ChatGPT uses to find and cite live sources, and ChatGPT-User fetches a page when a user asks about it directly. You can block GPTBot to keep your content out of model training while remaining fully visible and citable inside ChatGPT. Blocking OAI-SearchBot is what makes you invisible.

It depends entirely on which kind. Blocking training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) is a legitimate business decision with no visibility cost — publishers with licensable archives often should. Blocking answer engines (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot) removes your business from the assistants people increasingly use instead of search. For almost every commercial site trying to get found, answer engines should be allowed.

No. Google-Extended is a robots.txt token, not a crawler. It controls only whether content Googlebot already fetched may be used to train and ground Gemini. Google has stated it has no effect on Search ranking or on inclusion in AI Overviews. Disallowing Google-Extended is a safe way to opt out of model training without touching your search visibility.

Because robots.txt matching is per-group and most-specific-wins. If you have a User-agent: * group with Disallow: / but a separate, more specific group for that crawler, the crawler follows its own group and ignores the wildcard one entirely. The tool shows you which line decided each verdict so you can see exactly which group applied.

The major crawlers from OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity all publish their tokens and honour robots.txt. Some crawlers are widely reported to ignore it. robots.txt is a published instruction, not enforcement — if you need to guarantee a crawler cannot reach you, block it at the network or WAF layer. This tool reports what you are instructing crawlers to do, which is what determines the behaviour of every well-behaved one.

llms.txt is a proposed plain-text file at your site root that gives AI crawlers a curated map of your most important pages in a clean, readable form. It is not an official standard and no engine is required to read it. It costs almost nothing to publish and helps AI systems find your best content, so we treat it as a positive signal rather than a requirement.

It is free with no signup and no limit worth mentioning. We fetch three public files from the domain you enter — robots.txt, llms.txt, and the response headers of the page — and we do not log the domains checked or store results. The URL you enter is used for that request and nothing else.