What does a robots.txt tester do?
robots.txt is a plain-text file at the root of a hostname that tells crawlers which paths they may and may not request. It has no enforced syntax and no authority beyond convention — a crawler that ignores it will still fetch a blocked page — but every well-behaved search engine honours it, and a single stray line can therefore hide your most important pages from every one of them. This tester fetches the live file for the address you enter, works out which user-agent group applies to the crawler you chose, and reports whether the tested path is allowed or blocked and exactly which rule decided it.
The reason a separate tester is worth having is that robots.txt is read as a set of groups, and the group that applies to a crawler is not always obvious. A file with a `User-agent: *` block and a `User-agent: Googlebot` block means Googlebot is governed by one of them, not both. Records accumulate under whichever group came last, empty `Disallow:` values change the meaning of a group, and a rule such as `Disallow: /` inside a group that is not the one you think can block an entire site. Reading the file and reasoning about it is where mistakes happen; asking the file directly is not.
The tester reports a verdict for every group it finds, not just the one that applies, so you can see the shape of the whole file — which bots are blocked, which are given an exception, and which fall through to the wildcard. That is what makes the difference between knowing a page is allowed and knowing why.
How to test a URL against robots.txt
- Enter the full address of the page you care about, including the scheme and hostname — `https://example.com/private/page`. The path is what gets tested; the scheme and host are used to find the right robots.txt.
- Choose the crawler to test as. The preset list covers the ones that decide indexing — Googlebot, Bingbot, Twitterbot, facebookexternalhit — plus `*` for the wildcard group and this site's own crawler. Pick a specific bot rather than `*` when you are asking what a real search engine will do.
- If you are testing your own user agent, choose "a bot of my own" and type it exactly as it appears in the file's `User-agent` line, for example `AhrefsBot/7.0`. Matching is against the token, not the version.
- Press Test robots.txt. The file is fetched once from this site's server; nothing on the target site is crawled.
- Read the verdict at the top: allowed or blocked, for which crawler, and the exact rule that decided it — or a note that no rule matched and the default of allowed applies.
- Scan the group table underneath to see how the same path fares for every other crawler, which is how you catch a rule you meant for one bot that is quietly inherited by all of them.
robots.txt testing limits and rules
| Item | Limit |
|---|---|
| Fetches | robots.txt for the host of the URL you enter, at `/robots.txt` on the same scheme |
| Rule standard | RFC 9309 group selection: the most specific matching user-agent token wins, `*` is the fallback |
| Matching | Longest matching rule wins; on an equal-length tie, Allow beats Disallow; `*` and `$` wildcards are supported |
| Empty rules | An empty `Disallow:` is treated as a no-op and dropped from the effective rule set |
| Crawl-delay | Reported as written, and capped at 5 seconds when read by this site's own crawler |
| Missing file | A missing or unreachable robots.txt means everything is allowed, and the page says so rather than guessing |
| Echo of the file | The served text is shown up to 20,000 characters |
| Rate limit | 40 tests per minute per visitor |
| Not included | `noindex` meta tags and `X-Robots-Tag` headers, which are separate from robots.txt, and any crawling of the pages themselves |
| Not authoritative | This reports what the file says. A crawler may cache an older copy for up to a day, so a rule you just added may not take effect immediately. |
When to use it
- Checking whether a staging, admin or search-results path is actually blocked before you rely on the rule, and whether an `Allow` exception is overriding it.
- Confirming that a designed rule is not being inherited by Googlebot from a wildcard group you have since edited.
- Debugging a page that has vanished from search results — a stray `Disallow` is one of the few causes that leaves no other trace.
- Testing the effect of a new rule on an AI or third-party crawler by name, so an experiment does not accidentally block a search engine as well.
- Auditing a site you are about to migrate: reading every group tells you which crawlers the current owners intended to block and which they forgot.
Is it private?
The file has to be read by this site's server: a browser will not let a page fetch another origin's robots.txt cross-origin, so a purely in-browser tester could only ever read your own. When you press the button, your browser sends the address and the crawler name to this site's own endpoint, which requests `robots.txt` once with a crawler user agent, parses it and returns the result. The site you check sees one request from this server and never learns who asked. No cookies or credentials of yours are sent, only that single file is requested, and the address and file are not stored — the result is shown to you and dropped.
Frequently asked questions
The generator writes a new file. This tester reads a live one and answers a specific question about it: given this URL and this crawler, is it allowed, and which line said so. They are complements — you write with one, then check the result with the other — and the tester is the half that can tell you a file you wrote is having an effect other than the one you intended.
Look at which group applies. A `Disallow` written under one `User-agent` block applies only to that agent, so a rule you added for a specific bot will not block Googlebot. The group table spells this out: find the row highlighted as applying to your chosen crawler and see what it actually contains. The other common cause is ordering — records before the first `User-agent` line, or a group whose rules were broken by a blank line out of place.
No, and this is the trap that costs people months. `Disallow` stops a crawler from fetching the page, but a blocked URL can still be indexed if something links to it, and the crawler then cannot see a `noindex` tag to remove it, because it is not allowed to fetch the page carrying that tag. To keep a page out of results you want to allow crawling and use `noindex`; use `Disallow` for paths that should not be requested at all, such as search endpoints and duplicate filters.
The one you actually want to know about. Pick Googlebot to ask what Google's main crawler will do, `*` to ask what a crawler with no group of its own gets, or a named bot to check whether a third party is being blocked. The preset list is sorted so the search and social crawlers come first. The verdict for `*` and the verdict for a named bot can differ completely, and comparing them in the table is the point.
Crawl-delay asks a crawler to wait that many seconds between requests to your host. Googlebot ignores it, and Bing has its own setting, so it affects a minority of crawlers — but it is still used by some smaller engines and a great many scrapers, which is often the reason it is there. A large value slows legitimate crawling, so set one only if you have measured that automated traffic is actually hurting the server. This tool shows what the file asks for and caps it at 5 seconds when this site's own crawler obeys it.
Yes, by definition. A crawler that cannot read robots.txt — missing, empty, or answered with an error such as a 404 or 503 — treats the whole site as crawlable, because the file is a set of permissions rather than a list of protections. The tester reports the HTTP status alongside the verdict for exactly this case. If you expected rules to apply and the file is not being served, the file is not being read, and every rule in it is decoration.
Related tools
Written and maintained by Md. Nur Hossain Repon.