robots.txt Tester — see which line decides
Google retired its robots.txt tester and left you with a read-only file view. This one tests any path against any user-agent — including every major AI crawler — and names the exact group and line that allowed or blocked it.
Results for Googlebot
All 4 allowed/AllowedNo robots.txt rules loaded, so everything is allowed.
/blog/AllowedNo robots.txt rules loaded, so everything is allowed.
/wp-admin/AllowedNo robots.txt rules loaded, so everything is allowed.
/search?q=testAllowedNo robots.txt rules loaded, so everything is allowed.
The two rules that decide everything
Nearly every robots.txt bug comes from misunderstanding one of these. Both are implemented exactly as the major crawlers implement them, which is why this tool sometimes disagrees with a simpler checker.
A crawler reads exactly one group
Crawlers pick the group whose User-agent token most specifically matches their own name, then ignore every other group in the file — including User-agent: *. The wildcard group is a fallback for crawlers with no group of their own, not a set of defaults layered underneath.
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
# Googlebot crawls everything. It never reads the * group.
# Every other crawler is blocked from the whole site.The longest matching path wins, not the first
Order in the file is irrelevant. The rule with the most characters in its matching path takes precedence, and when an Allow and a Disallow tie at the same length, Allow wins. This is what makes the standard WordPress exception work.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# /wp-admin/admin-ajax.php → Allowed (28 chars beats 11)
# /wp-admin/options.php → BlockedFrequently asked questions
Google removed the robots.txt tester from Search Console — is this the same thing?
It does the same job. Google retired the robots.txt Tester report in Search Console at the end of 2023, leaving only a read-only view of the fetched file with no way to test URLs against it. This tool restores that: pick a user-agent, enter any set of paths, and see which specific line decides each one. It also tests user-agents Google’s tool never covered, including every major AI crawler.
How does robots.txt decide which rule wins when several match?
Two rules, both implemented here. First, group selection: a crawler follows the group whose User-agent token most specifically matches its own name, and ignores every other group including the wildcard one. So a Googlebot group means Googlebot never reads the User-agent: * group at all. Second, within that group, the rule with the longest matching path wins — not the first one listed. If an Allow and a Disallow match at exactly the same length, Allow wins.
Do wildcards and $ work in robots.txt?
They are not in the original specification but every major crawler supports them, and RFC 9309 codified the behaviour. An asterisk matches any sequence of characters, so Disallow: /*?sort= blocks every URL containing that query parameter. A dollar sign anchors the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf?download=1. This tester evaluates both.
Does Disallow stop a page from appearing in Google?
No — and this trips people up constantly. Disallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still list it, showing the URL with no description. To keep a page out of results you must let crawlers reach it and serve a noindex meta tag or X-Robots-Tag header. Blocking it in robots.txt actively prevents Google from seeing the noindex you added.
Can I test a robots.txt file before publishing it?
Yes. Switch to "Paste robots.txt", drop in your draft and test paths against it. That mode runs entirely in your browser — nothing is sent to us or anywhere else — so you can validate a change safely before it goes live.
Why does my Disallow: / not block the crawler I tested?
Almost always because that crawler has its own more specific group somewhere else in the file. A crawler reads exactly one group: the most specific match for its token. If you have both a wildcard group with Disallow: / and a Googlebot group with Allow: /, Googlebot follows its own group and crawls freely. The reason line under each result names the group that applied.
One wrong line can cost a whole site.
A stray Disallow: / left over from a staging deploy is one of the most common causes of a traffic collapse we get called in to diagnose. A technical audit finds it in minutes.