Randozirobots.txt URL Tester
A single wrong line in robots.txt can hide a whole folder from search engines. Paste the file, list the pages you care about and see the verdict for each, with the rule that decided it.
Allowed
4
Blocked
2
Robot
Googlebot
| Address | Verdict | Deciding rule |
|---|---|---|
| https://example.com/ | Allowed | No rule matches |
| https://example.com/admin/settings | Blocked | Disallow: /admin/ (line 2) |
| https://example.com/admin/help | Allowed | Allow: /admin/help (line 4) |
| /files/report.pdf | Blocked | Disallow: /*.pdf$ (line 3) |
| /files/report.pdf?download=1 | Allowed | No rule matches |
| /robots.txt | Allowed | /robots.txt is always allowed |
How it works
- Paste the contents of your robots.txt into the first box.
- Type the crawler's name, for example Googlebot, or * to test the rules for every robot.
- List the addresses or paths to check, one per line. A full URL or just /path both work.
- Read the verdict for each line, with the rule and its line number, then fix the file and test again.
What people use it for
Before a site launch
Check that the pages that should rank are open and the admin or staging folders are not.
After a migration
Paste the list of old and new URLs and catch the ones a new Disallow line blocks by accident.
Understanding a conflict
See why an Allow line wins over a Disallow, or the other way round, instead of guessing.
Questions
- How is the winning rule chosen?
- Following RFC 9309, section 2.2.2: among the Allow and Disallow rules that match the path, the one with the most characters wins. If an Allow and a Disallow match with the same length, Allow wins. If nothing matches, the page is allowed.
- What do * and $ do?
- In a rule, * stands for any run of characters and a final $ means the path must end there. So /*.pdf$ blocks /files/a.pdf but not /files/a.pdf?download=1. Path matching is case sensitive: /Admin and /admin are different.
- Which group applies to my crawler?
- The groups that name your crawler (the name is not case sensitive) are merged and used. If no group names it, the group for * is used. A group for a specific crawler replaces the * group completely; the two are not added together. An empty Disallow line blocks nothing, and /robots.txt itself is always allowed.
- Does this match what real crawlers do?
- It follows the RFC 9309 standard; a given crawler can differ in small details. It does not fetch your live file, ignores the host name of the URLs you paste, and does not know about the HTTP status your server returns for /robots.txt, which also changes what crawlers do. To keep your browser responsive it reads up to 1,000 rules of at most 500 characters and 5 * wildcards, and addresses up to 300 characters; anything beyond is reported as left out. Only the first 500 KiB of the file are read, the minimum a crawler must parse.