Robots.txt generator
Build crawl rules and sitemap declarations from a form.
Your inputs
Your results
Ready when you areYour results will appear here.
How it works
Generate one user-agent group with explicit Allow/Disallow paths and absolute sitemap declarations, rejecting line-injection and comment ambiguities.
Read the source reference ↗A few things to know
Robots rules do not secure private pages or guarantee exclusion from search. One group only; unsupported crawler-specific directives are omitted. Review path specificity with the separate rule tester.
When to use this tool
Create a small, readable robots.txt from explicit paths. The file belongs at the root of the correct scheme, host and port.
- Prepare a launch file.
- Declare a sitemap.
- Document a small crawl policy.
How to use robots.txt generator
Prepare the inputs
Choose one user-agent token.
Choose the settings
Enter encoded allow/disallow paths and sitemap URLs.
Review and export
Download robots.txt, publish at the origin root and test the rules.
Try the supplied example
Inputs
agent: *; allow: /; disallow: /admin/ /internal/; sitemaps: https://example.com/sitemap.xml
Result
The example allows the root, disallows admin and internal paths and declares one sitemap.
What it means
Use authentication for privacy and an appropriate noindex method for indexing control.
Common mistakes to avoid
Crawl rules are not authentication
Crawl rules are not authentication.
Blocking crawl is not a dependable noindex method
Blocking crawl is not a dependable noindex method.
Frequently asked questions
Definitions, edge cases, and practical next steps for this tool. Search all tool FAQs ↗
Does robots.txt prevent indexing?
Not reliably. A blocked URL can still be discovered and listed without its content.
Does it protect private content?
No. Use authentication and access controls.
Where should it be published?
At /robots.txt on the relevant origin.
Can I add a sitemap?
Yes. Supply absolute HTTP or HTTPS sitemap URLs.
Can I make several crawler groups?
This version generates one group. Edit a full file deliberately if more groups are needed.
Are path matches case-sensitive?
Path matching follows crawler rules; check exact capitalization and specificity with a tester.
Are my inputs uploaded or stored?
Processing happens locally in your browser. This tool does not upload inputs. Downloads stay on your device; sharing or opening them in another service is your choice.
What should I check before using the output?
Robots rules do not secure private pages or guarantee exclusion from search. One group only; unsupported crawler-specific directives are omitted. Review path specificity with the separate rule tester.
Explore the topic in the blog
Sources and method
Use the references below to check definitions and platform guidance. The model’s scope and limits are explained above. Read our editorial standards.