robots.txt Generator

Build a robots.txt file. Pick a preset for your platform, add your sitemap, and choose whether to block SEO crawlers or the AI training bots that have appeared in the last few years.

The output is ready to save as robots.txt in your site root — it must be at the root, since crawlers look nowhere else.

How it works

robots.txt tells crawlers which parts of a site they may request. It is a voluntary convention, not an access control: well-behaved crawlers honour it, and anything malicious ignores it entirely. Never use it to protect anything sensitive — you are publishing a list of the paths you would rather people did not visit.

The single most misunderstood point: Disallow does not remove a page from search results. It stops the page being crawled, not indexed. Google can still list a URL it has never fetched — showing the bare URL with no description — if other sites link to it. Worse, because it cannot crawl the page, it will never see a noindex tag there. To keep something out of the index, allow crawling and use noindex; to keep crawlers off a heavy path, use Disallow.

Other things worth knowing:

• It must live at the domain root — /robots.txt. A file at /blog/robots.txt is ignored.
• Rules are per host and protocol. https://example.com and https://www.example.com need their own files.
• A crawler follows the single most specific matching User-agent block, and ignores the rest — so a named block must repeat any rules from * that should still apply to it.
• Blocking CSS and JavaScript breaks Google's ability to render your pages and can materially hurt rankings. This was common advice years ago; it is now actively harmful.

On AI crawlers: GPTBot, ClaudeBot, Google-Extended and others honour robots.txt for training data collection. Blocking them is a policy decision, not a technical one — and note that blocking Google-Extended affects AI training only, not your normal Google Search ranking.

Frequently asked questions