robots.txt in 2026: What Changed, and What You Should Actually Block

Robots.txt is the smallest file on your site and the one most capable of destroying it. Three characters in the wrong place will remove a company from Google, and the file has no validation, no error log, and no warning before it takes effect.

It also does far less than people believe. A large share of what gets written into robots.txt is solving a problem the file cannot solve, and a fair amount of it is causing new ones.

Here is what belongs in the file in 2026.

---

What the File Does, Precisely

Robots.txt tells cooperating crawlers which URLs they may request. That is the entire function.

Two consequences follow, and both surprise people.

Disallow does not mean deindex. If a page is blocked in robots.txt but linked from elsewhere, Google can still index the URL. It cannot read the page, so it shows the URL with no description, sometimes with a snippet pulled from the anchor text linking to it. The classic disaster is blocking a directory to remove it from search and finding those URLs still listed months later, now with no title and no content.

To remove a page from the index you need a noindex meta tag or header on the page itself. Which means the crawler has to be allowed to fetch it. Disallow and noindex on the same URL cancel each other out, because the crawler that would read the noindex is not permitted to arrive.

Compliance is voluntary. Google, Bing, and every AI operator with a legal department honour it. Scrapers do not. If your requirement is enforcement rather than preference, this is the wrong tool and you want authentication or firewall rules.

What Actually Belongs in Disallow

Short list. Most sites need fewer lines than they have.

Faceted navigation and filter parameters on e-commerce or listing sites. A product catalogue with four filters can generate hundreds of thousands of URL combinations from a few hundred products. This is the one case where crawl waste is genuinely expensive, and it is worth getting right.

Internal site search results. /?s= on WordPress, /search?q= on most other systems. These are infinite, thin, and occasionally get indexed with embarrassing queries in the title tag.

Cart, checkout, account and admin paths. No search value, and crawlers hitting add-to-cart URLs cause real problems in analytics.

Genuinely duplicate print or export views, where a canonical tag is not practical.

That is close to the complete list for a normal business site. If your robots.txt is thirty lines long, most of those lines are either doing nothing or doing harm.

What You Must Never Block

CSS and JavaScript files. This is the most damaging common mistake and it persists because it was standard advice in 2010. Google renders pages to evaluate them. Blocking the resources needed to render means Google sees a broken layout and cannot verify the page is mobile-friendly. Any Disallow: /wp-includes/, Disallow: /assets/ or Disallow: *.js$ line in your file should come out today.

Images that appear in your content, unless you specifically want them out of Image Search, which for most local service businesses is free traffic they are choosing to decline.

Your sitemap file. Obvious once stated, and we still find it.

Anything you want removed from the index. See above. Allow it, add noindex, wait for the recrawl, then block it later if you want to save the crawl.

The AI Crawler Section

This is the part of the file that has actually changed, and where the most damage is currently being done.

There are now three distinct categories of AI user-agent, and treating them as one thing is what causes the harm. Training crawlers like GPTBot, ClaudeBot and CCBot collect data for model training, and blocking them costs no visibility. Answer crawlers like OAI-SearchBot, PerplexityBot and Claude-SearchBot are how assistants find and cite you, and blocking them removes you from the answer. Search crawlers feed both classic results and AI Overviews, and should never be blocked at all.

The pattern we find repeatedly is a file that blocks all three because it was copied from a 2024 article about protecting content from AI. We wrote the full decision framework in is ChatGPT allowed to crawl your website, including the configuration we recommend and how to check what your site currently allows.

The short test: run your domain through the AI Bot Access Tester and look only at the answer group. Anything disallowed there is costing you citations right now.

Worried your robots.txt is quietly blocking something important? Get a Free Visibility Audit →

Syntax Rules That Catch People Out

The parsing rules are not intuitive and getting them wrong is how the destructive mistakes happen.

Only the most specific matching user-agent group applies. If you have a User-agent: * group and a User-agent: Googlebot group, Googlebot obeys its own group and ignores the wildcard group entirely. Rules do not accumulate across groups. A Disallow: /admin/ in the wildcard block does not apply to Googlebot if a Googlebot block exists anywhere in the file.

Within a group, the most specific path rule wins, not the first or last one. Allow: /blog/ beats Disallow: /b for the URL /blog/post/, because the matching prefix is longer.

A trailing slash changes the meaning. Disallow: /print blocks /print, /printing, /printers-in-toronto. Disallow: /print/ blocks only the directory. The version without the slash has taken out entire content sections by accident.

Wildcards are supported by Google and Bing, but not universally. * for any sequence and $ for end-of-URL both work with the major crawlers. Smaller crawlers may treat them as literal characters.

The file must be at the root. /robots.txt, per protocol and per subdomain. blog.example.com needs its own; the root domain's file does not govern it.

Because none of this reads intuitively, test rather than reason. Paste your file into the robots.txt Tester with the URLs you care about and the user-agents you care about, and it reports which specific line allows or blocks each combination. Do this before deploying, not after.

The Sitemap Directive

Add it. One line, absolute URL, and it works regardless of which user-agent group it appears in:

Sitemap: https://example.com/sitemap.xml

It is not a substitute for submitting the sitemap in Search Console, and it does not help Google much once your site is established. It does help every other crawler that will never have a verified property on your domain, which now includes several AI operators that discover content through sitemaps.

If your sitemap is out of date or full of URLs that redirect, 404 or canonicalise elsewhere, pointing crawlers at it makes things worse rather than better. Our Sitemap Generator crawls the site, honours your robots.txt, and drops every URL that would be rejected, which usually produces a considerably shorter file than whatever your CMS plugin is publishing.

Crawl Budget: Mostly Not Your Problem

A great deal of robots.txt advice is written for sites with millions of URLs and then applied to sites with fifty pages.

If your site is under a few thousand URLs and reasonably fast, Google crawls it fully and crawl budget is not a constraint you need to manage. Trimming crawl paths will not improve your rankings, and the effort spent on it is better applied to almost anything else.

Crawl budget starts to matter at large scale, or when a site generates unbounded URL space through parameters or a calendar, or when the server is slow enough that Google throttles its own crawl rate. Search Console's Crawl Stats report tells you which situation you are in. Check it before optimising anything, because the common outcome is discovering there was nothing to fix.

A Working Example

For a typical WordPress service business:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Nine lines. No CSS or JS blocks, no attempt to deindex anything, answer crawlers left alone, and one deliberate training-crawler decision that the owner can reverse in ten seconds if they change their mind.

Most sites we take over are running four times that length and getting less out of it.

Before You Deploy Anything

Read the current file first and find out why each line is there. A Disallow you do not understand is more likely to be load-bearing than obsolete, and removing it blind is how staging sites end up indexed.

Then test the new version against real URLs, deploy, and check Search Console's coverage report a week later for pages that have moved into "Blocked by robots.txt". That report is the only feedback loop this file has.

The rest of the crawl and index stack is covered in our technical SEO guide, and if you want to see what crawlers actually receive once they are allowed through, what Googlebot actually sees covers the next layer down. Google's own robots.txt specification is the authoritative reference for the parsing rules above.

Not certain what your robots.txt is doing to your visibility?

A Free Visibility Audit covers crawler access across search and AI agents, indexation coverage, and the specific directives costing you pages.

Get a Free Growth Audit →

---

Related reading: What Googlebot Actually Sees | Is ChatGPT Allowed to Crawl Your Website? | The Complete Technical SEO Guide