Generate robots.txt and a URL-List Sitemap for a Small Site
Prepare crawl rules and sitemap XML without confusing discovery, indexing and access control.
Write the intended public inventory
List canonical pages that should be available to visitors and distinguish them from authenticated or internal routes. The URL-list generator does not crawl the site or discover its pages. Use an export or a reviewed manual list and keep it with the launch notes. A page that redirects, returns an error or carries noindex needs a separate review before it belongs in an indexable sitemap.
Create a readable robots group
The robots generator example sets User-agent: *, allows /, disallows /admin/ and /internal/, and declares https://example.com/sitemap.xml. It produces one group with explicit paths. Publish the file at /robots.txt on the intended scheme, host and port. Path specificity still matters, so test representative allowed and blocked paths with the existing rule tester before relying on the generated file.
Build the two-page sitemap example
Paste https://example.com/ and https://example.com/about/ into the URL-list generator. It produces two URL entries in the standard sitemap namespace. Duplicate serialized URLs are removed, fragments are rejected and this version requires one origin. Query ampersands are escaped as XML without changing the underlying URL. Leave the optional shared lastmod blank unless a meaningful page update date genuinely applies to every listed page.
| Choice | Primary purpose | Limit |
|---|---|---|
| robots.txt | Crawl instructions | Does not secure or reliably deindex a URL |
| sitemap.xml | Canonical URL discovery | Does not guarantee crawling or indexing |
| Authentication | Restrict access | Requires real server or application enforcement |
| noindex | Request exclusion from indexing | Crawler must be able to read the instruction |
The generators do not inspect live pages or configure the server.
Use this example Opens Robots.txt with these inputs.
Choose the right control
A robots disallow controls crawling, not authentication, and a blocked URL may still be listed in search. A sitemap helps discovery; it does not override crawl restrictions or establish indexability. Use access controls for private content and a suitable noindex method for indexing decisions. Do not put sensitive route information in a public robots file under the assumption that the file hides it.
Check the actual published responses
Download both files, publish them in the intended locations and retrieve those exact URLs. Confirm the host, sitemap links, response content and referenced page behavior. Test representative crawl rules again using the deployed file, rather than the draft. If the URL inventory becomes large, use the existing sitemap validator and splitter to review published XML and protocol bounds. Keep launch intent and observed responses as separate evidence.