Drag-n-Drop
to upload your archive
Robots.txt Generator
Build a valid robots.txt in seconds. Free, no signup
Pick a ready-made template, add your own Allow and Disallow rules, drop in your sitemap URL, and the file is written for you line by line as you type. Everything runs in your browser, so nothing about your website is ever sent to us. Copy it or download a ready-to-upload robots.txt.
Crawl rules
| User-agent | Directive | Path | Actions |
|---|
Leave the path empty for a bare Disallow:. That is the canonical way to say “crawl everything”. Use / to block the whole website.
Optional directives
Honored by Bing and Seznam. Google and Yandex ignore it; both set crawl rate automatically, and you can slow Bingbot in Bing Webmaster Tools.
A legacy Yandex directive; Yandex stopped reading it in 2018. Leave it blank unless a niche crawler you rely on still parses Host; use 301 redirects and canonical tags to signal your preferred domain instead.
Must be absolute URLs. They are written at the very bottom of the file, outside every user-agent group, which is the convention crawlers expect.
Emits Content-Signal: search=yes, ai-train=no inside each group: “you may index me, you may not train on me”. It is a declaration of intent, not an enforcement mechanism, and it must stay inside the group because a blank line would end the block. Static.app publishes its own Content-Signal line (ours opts in to AI use) at /robots.txt.
-
Correct grouping, automatically
Rules for the same user-agent are merged into one block, blocks are separated by a blank line, and Sitemap lines go last. The three things hand-written robots.txt files get wrong most often.
-
One-click AI crawler block
Twenty named AI and LLM agents (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider and more) blocked in a single click, without touching Googlebot or Bingbot.
-
Live preview, live warnings
The file rewrites itself on every keystroke, and the status line flags the classics: a path missing its leading slash, a relative sitemap URL, an empty user-agent.
How to create a robots.txt file
-
1
Pick a preset, then add or remove rules until the crawl rules match your website.
-
2
Paste your sitemap URL and, if you need it, a crawl delay.
-
3
Download the file and upload it to your website root, so it answers at /robots.txt.
Robots.txt example: allow everything, block AI training bots
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlSearch crawlers may fetch every page, GPTBot may fetch none, and the sitemap line points every crawler at your URL list. Pick “Block AI crawlers” above to generate the full version with all twenty AI agents.
What a robots.txt file actually controls
-
Crawling, not indexing
Disallow tells a crawler not to fetch a URL. It does not remove that URL from search results; a blocked page can still be listed if other websites link to it. To keep a page out of the index, let it be crawled and serve a
noindexmeta tag or header. -
Crawl budget on big sites
Faceted navigation, internal search results and endless filter combinations can burn a crawler's time on URLs that will never rank. Blocking those patterns points the budget at the pages you actually care about. On a small brochure website this matters far less than people think.
-
A public, voluntary standard
Anyone can read your robots.txt, and compliance is opt-in. Well-behaved crawlers follow it; scrapers and spam bots ignore it entirely. Never list secret paths there, and never treat it as access control; use authentication for anything that must stay private.
Got your robots.txt? Publish the whole website with free static hosting
Select File
or drop your archive to upload
Frequently asked questions
What is a robots.txt file?
robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your website they may request. It is the first thing most well-behaved bots fetch. It is a set of instructions, not a security boundary: it is publicly readable and hostile scrapers simply ignore it, so never use it to hide sensitive URLs.
Do I need a robots.txt file?
No. Without one, crawlers assume everything is allowed, which is fine for most small websites. Add one when you need to keep bots out of specific paths, block AI crawlers, or point crawlers at your sitemap; and once you do, keep it valid, because a malformed file can accidentally block more than you intended.
What is the difference between Allow and Disallow?
Disallow blocks matching paths; Allow carves an exception out of a broader Disallow. The classic pair is Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php. When rules conflict, the most specific one (the longest matching path) wins.
Where does robots.txt go?
It must live at the root of the host, at https://example.com/robots.txt, and it only applies to that exact host and protocol. A file at /blog/robots.txt does nothing, and a file on www.example.com does not cover example.com.
Does robots.txt stop a page from being indexed by Google?
No. Disallow stops a crawler from fetching a URL; it does not remove that URL from search results. A blocked page can still be indexed from external links, appearing with no snippet. To keep a page out of the index you need a noindex robots meta tag, and blocking that page in robots.txt at the same time is self-defeating, because the crawler can never fetch the page to see the noindex.
How do I block AI crawlers?
Add a Disallow rule for each AI user agent you want to exclude. The "Block AI crawlers" preset here covers the current major ones, including GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent. Note that Google-Extended controls AI training use without affecting normal Google Search crawling, which is usually what website owners actually want.
What is the Content-Signal line?
Content Signals is a newer, machine readable way to express intent per use case rather than per bot, for example allowing search indexing while declining AI training. It does not replace Disallow rules and adoption is still uneven, so treat it as a clear statement of intent that sits alongside your explicit rules.
Should I list my sitemap in robots.txt?
Yes. A Sitemap: line with the full absolute URL is read by every major engine and costs nothing. It is the simplest way to make sure crawlers find your sitemap even if you never submit it manually. Build one with our XML Sitemap Generator.
Does crawl-delay work with Google?
No. Google ignores Crawl-delay entirely and adjusts its crawl rate automatically; Yandex dropped support in 2018. Bing does honor it. Only add it if a specific bot is genuinely overloading your server, and prefer fixing the underlying performance problem instead.
How do I test my robots.txt?
Upload the file, open it in a browser to confirm it is served as plain text at the root, then use the robots.txt report in Google Search Console to see how Google parses it. If you host with Static.app, dropping robots.txt into your project root is all that is needed: it is served as a static file, no configuration required.