Robots.txt Tester
Test whether a URL is allowed or blocked by a robots.txt file, applying Google's documented matching rules.
/admin/secret.html with user-agent *| Matched group | * |
|---|---|
| Matched rule | /admin/ (disallow) |
| Explanation | The Disallow rule /admin/ matches and no Allow rule overrides it. |
{
"groups": [
{
"userAgent": "*",
"allow": [
"/admin/public.html"
],
"disallow": [
"/admin/",
"/private/",
"/*.pdf$"
]
},
{
"userAgent": "googlebot",
"allow": [
"/"
],
"disallow": []
}
],
"sitemaps": [
"https://utilvia.online/sitemap.xml"
]
}
Try these URLs
How robots.txt matching works
- User-agent — identifies the crawler.
*matches all crawlers not covered by a specific rule. - Disallow — blocks matching paths.
- Allow — overrides Disallow for matching paths.
- Sitemap — declares the sitemap location.
- * — wildcard matching any sequence of characters.
- $ — matches end of URL (path + query).
- When Allow and Disallow match with equal length, Allow wins. Otherwise, the most specific (longest) rule wins.
How it works
1. Paste your robots.txt — Copy the content of your robots.txt file into the text area, or load the sample to experiment.
2. Enter a URL to test — Type the path you want to check (e.g., /admin/users or /products/item-123).
3. Specify user-agent — Enter a crawler name (e.g., Googlebot, * for all) to see which rule group applies.
4. Run the test — Click "Test" to see if the URL is allowed or blocked, which rule matched, and why.
5. Review the explanation — The tool shows the matched group, the specific rule, and a plain-English explanation of the logic.
Examples
Example 1: Blocking admin areas
User-agent: * Disallow: /admin/ Allow: /admin/public.html
Test /admin/secret → Blocked (Disallow matches)
Test /admin/public.html → Allowed (Allow overrides for this specific file)
Example 2: Wildcard patterns
User-agent: * Disallow: /*.pdf$
Test /docs/report.pdf → Blocked ($ matches end of URL)
Test /docs/report.pdf?version=2 → Allowed (query string means URL doesn't end with .pdf)
Example 3: Specific crawler rules
User-agent: * Disallow: /private/ User-agent: Googlebot Allow: /
Test /private/data with user-agent * → Blocked
Test /private/data with user-agent Googlebot → Allowed (specific group takes precedence)
Frequently asked questions
What matching rules does this tool use?
The tool implements Google's documented robots.txt matching rules: longest match wins, wildcards (*) match any sequence, $ matches end of URL, and Allow overrides Disallow when lengths are equal.
Can I test my live robots.txt file?
Due to browser CORS policies, the tool cannot fetch robots.txt from most websites directly. Copy and paste the content manually, or use a browser extension that bypasses CORS.
What's the difference between Allow and Disallow?
Disallow blocks access to matching paths. Allow permits access. When both match a URL, the longer (more specific) rule wins. If lengths are equal, Allow wins.
What does the * wildcard do?
* matches any sequence of characters. Disallow: /admin/* blocks everything under /admin/. Disallow: /*.pdf$ blocks all PDF files.
What does the $ anchor do?
$ matches the end of the URL path (including query string). Disallow: /*.pdf$ blocks URLs ending in .pdf but allows /file.pdf?download=1.
Do I need a robots.txt file?
It's recommended. Without one, crawlers assume they can access everything. A robots.txt file gives you control and can reduce unnecessary crawler traffic.
Will this tool prevent my site from being indexed?
No — robots.txt controls crawling, not indexing. To prevent indexing, use meta robots tags (<meta name="robots" content="noindex">) or X-Robots-Tag headers.