Robots.txt Generator: Create a Robots.txt File in Seconds
Pick a preset, choose which bots may crawl which folders, add your sitemap and copy the file. A built-in tester shows whether Googlebot can reach any URL before you upload anything.
A preset only fills in the rules below. You can change every line afterwards.
One path per line, starting with /. You can use * as a wildcard and $ for the end of a URL, for example /*.pdf$. Leave both boxes empty to allow everything for that group.
Upload the file to the root of your site so it opens at https://yourdomain.com/robots.txt. A robots.txt in a sub-folder is ignored.
The tester follows Google's matching rules: the longest matching rule wins, and Allow wins a tie. Other crawlers can differ slightly.
How to create a robots.txt file
- Choose a preset, or start with “Allow everything”.
- Under Rules, list the folders you do not want crawled, for example
/wp-admin/or/cart/. Add an Allow line for any file inside a blocked folder that must stay reachable. - Add your sitemap address so crawlers find all your pages.
- Test a few important URLs. Make sure your key pages, CSS and JavaScript files are allowed.
- Copy the file or download it and upload it to your site root. On WordPress with Rank Math or Yoast you can paste it into the plugin's robots.txt editor instead.
Robots.txt directives explained
| Directive | What it does | Example |
|---|---|---|
User-agent | Names the crawler the rules below apply to. * means every crawler that has no group of its own. | User-agent: Googlebot |
Disallow | Asks the crawler not to fetch URLs that start with this path. An empty value allows everything. | Disallow: /cart/ |
Allow | Re-opens a path inside a blocked folder. | Allow: /wp-admin/admin-ajax.php |
Sitemap | Gives the full address of an XML sitemap. It applies to the whole file, not to one group. | Sitemap: https://example.com/sitemap.xml |
Crawl-delay | Asks a crawler to wait between requests. Google ignores it. Bing and some others follow it. | Crawl-delay: 10 |
What robots.txt cannot do
- It does not remove pages from Google. A blocked URL can still appear in results if other sites link to it. To keep a page out of search, leave it crawlable and add a
noindextag. Google no longer supportsnoindexinside robots.txt. - It is not security. The file is public and only polite crawlers follow it. Protect private areas with a login or server rules.
- Blocking AI crawlers is a request, not a lock. Crawlers from the major AI companies say they follow robots.txt, but the file cannot force any bot to obey.
Common mistakes
- Leaving
Disallow: /on after launch. This blocks the whole site. It is a common staging leftover. - Blocking CSS and JavaScript. Google needs them to render your pages the way visitors see them.
- Wrong case. Paths are case sensitive.
/Photos/and/photos/are different. - Missing trailing slash.
Disallow: /shopalso blocks/shop-newsand/shopping. Use/shop/to block only the folder. - Several groups for the same bot. Crawlers merge them, which can give results you did not expect. Keep one group per bot.
Related tools
Robots.txt Checker · Sitemap Checker · Redirect Generator · Redirect Checker · Meta Tag Analyzer
Frequently asked questions
Where do I put the robots.txt file?
In the root folder of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file. The file must be plain text in UTF-8.
Does my website need a robots.txt file?
No. Without one, crawlers assume everything can be crawled. A file is useful when you want to keep crawlers out of some folders, point to your sitemap, or manage particular bots.
How do I block ChatGPT, Claude or other AI crawlers?
Use the “Block AI training bots” preset. It adds a group with the crawler names used by OpenAI (GPTBot), Anthropic (ClaudeBot), Common Crawl (CCBot), Google-Extended, Applebot-Extended and Bytespider, and disallows the whole site for them. This does not affect your Google Search ranking, and compliance by the bots is voluntary.
What is the difference between Allow and Disallow?
Disallow asks crawlers to skip a path. Allow makes an exception inside a blocked path. When both match a URL, Google uses the longer, more specific rule.
How large can a robots.txt file be?
Google reads up to 500 KiB and ignores anything after that. Keep the file short and use wildcards instead of long lists of URLs.
I blocked a page but it still shows in Google. Why?
Robots.txt controls crawling, not indexing. Google can list a blocked URL without a description if other pages link to it. Allow crawling and use a noindex tag, or remove the page with a password or a 410 status.
About this tool. The generator writes standard robots.txt syntax as described in Google's documentation and the Robots Exclusion Protocol (RFC 9309). The tester applies the same longest-match rule that Google documents. We checked the output against presets, wildcard and end-of-URL patterns, ties between Allow and Disallow, and Googlebot-Image falling back to Googlebot. Everything runs in your browser and nothing you type is sent to our server.
Last reviewed: October 2026.