Robots.txt rules tester
Paste robots.txt and test whether specified URLs are allowed for a crawler.
Your inputs
Your results
Ready when you areYour results will appear here.
How it works
Inspect pasted robots.txt rules before deployment. The tester chooses the longest matching crawler product token, combines groups with equal specificity, and falls back to * only when no specific group matches. Allow and Disallow match case-sensitive paths plus query strings. Asterisk matches any sequence, a final dollar sign anchors the end, longer matching normalized rules take precedence, and Allow wins equal-length conflicts. Each result names the winning rule and its source line.
Read the source reference ↗A few things to know
A local Google-style rule test, not a live crawl or indexing report. Other crawlers may interpret directives differently. Product-token matching supports exact tokens and hyphen/underscore descendants, not arbitrary substrings of browser headers. Unsupported fields and malformed paths are ignored with warnings; undocumented Google typo recovery and index.html Allow normalization are not simulated. Percent escapes are normalized, unreserved escaped characters are decoded, raw non-ASCII characters are encoded, and encoded reserved separators remain distinct. URL parsing normalizes dot segments. This tool does not simulate fetch errors, redirects, cache history, HTTP status handling, server routing or encoding errors. Sitemap entries are informational. Robots.txt is scoped to a single protocol, host and port; it cannot enforce confidentiality or guarantee deindexing.
When to use this tool
A small robots.txt edit can affect many URLs, especially when crawler-specific groups, wildcards and query strings interact. This tester evaluates pasted rules against a list of paths or URLs and names the winning rule. It uses a practical Google-style interpretation to support review before deployment. The result describes the supplied rules, not a live crawl, server access policy or confirmation that a page is indexed.
- Test a proposed Disallow rule against important pages before publishing.
- Check a specific Allow exception within a blocked directory.
- Compare crawler-specific rules with the fallback wildcard group.
How to use robots.txt rules tester
Paste the rules and crawler token
Enter robots.txt contents and a crawler product token such as Googlebot or Bingbot. Use the product token rather than a complete browser user-agent string. An empty file permits crawling under this test.
Supply scope and test URLs
Optionally enter the root robots.txt URL to check protocol, host and port scope. Add one HTTP(S) URL or path per line, including query strings when relevant. You can test up to 500 entries.
Review winning rules
Run the test and inspect Allowed, Blocked and invalid or outside-scope results. Each match identifies a source line. Check warnings and your real server behavior separately before deploying the edited file.
Test an exception and a file suffix rule
Inputs
Use the supplied Googlebot rules blocking /private/, allowing /private/help$ and blocking /*.pdf$. Test the five example URLs with the example.com robots scope.
Result
Three URLs are allowed and two blocked. /private/account and /report.pdf are blocked. /private/help, /report.pdf?download=1 and /Private/account are allowed.
What it means
The exact help exception wins, the final $ makes the PDF rule stop before a query-bearing variant, and uppercase Private differs from lowercase private. Small pattern details change the outcome.
Common mistakes to avoid
Using robots.txt as access control
Crawl directives do not protect confidential content. Use authentication or server authorization when people or software must not access a resource.
Assuming a block removes a search result
Blocking a crawler is not a reliable removal instruction. Consider the actual indexing requirement and official guidance instead of equating blocked crawl with deindexing.
Forgetting query strings or case
Matching includes the path and query string and is case-sensitive. Test representative variants, especially when a pattern ends with $ or a site exposes mixed-case routes.
Frequently asked questions
Definitions, edge cases, and practical next steps for this tool. Search all tool FAQs ↗
How does the tester choose a crawler group?
It selects the longest matching supported crawler product token and merges equally specific groups. The wildcard group is used only if no specific group matches. Enter a token such as Googlebot-News rather than an arbitrary full user-agent header for a meaningful test.
Which rule wins when Allow and Disallow both match?
Longer matching normalized rules take precedence. When specificity is equal, Allow wins. The result names the winning rule and line so you can examine that interaction instead of treating the file as a simple first-match list.
What do the asterisk and dollar sign mean?
An asterisk matches a sequence of characters. A final dollar sign anchors the match to the end of the compared path plus query. Consequently, /*.pdf$ can block /report.pdf while allowing /report.pdf?download=1 under the supplied rules.
Why is a URL outside the robots.txt scope?
The optional robots URL defines a protocol, host and port scope. A rule file for https://example.com does not automatically govern http://example.com or another subdomain. The tester marks those differences so you do not apply one site’s rules to another origin.
Does the Sitemap directive allow crawling?
No. Sitemap entries are informational in this tester and do not override matching crawl rules. The tool displays them but does not fetch the sitemap. Test the sitemap’s listed page URLs against the relevant rules when checking a proposed change.
Does Allowed mean Google will index the page?
No. It means the pasted rules permit that request under this interpretation. Server responses, access restrictions, canonicalization, content and other indexing signals are outside the test. It neither crawls the page nor checks Search Console indexing status.
Are percent-encoded and non-ASCII paths supported?
The tester normalizes percent escapes, decodes unreserved escaped characters and encodes raw non-ASCII text. Encoded reserved separators remain distinct. URL parsing can also normalize dot segments. Review edge cases against the actual crawler and server rather than assuming all spellings are equivalent.
Can I use this to check Bing or every crawler?
You can enter other crawler tokens, but the matching behavior is a practical Google-style model. Other crawlers may handle directives differently. It does not simulate fetching errors, redirects, caching or every undocumented recovery rule; use the relevant crawler’s documentation for final review.
Explore the topic in the blog
Sources and method
Use the references below to check definitions and platform guidance. The model’s scope and limits are explained above. Read our editorial standards.