Map Extract
Free tool

Robots.txt generator

Write the rules once, group by group: which crawler, which paths to keep out, which exceptions to let through. Add your sitemap addresses and, if you want, a block for five AI training crawlers, then download a ready robots.txt.

Runs in your browser. Nothing you enter is uploaded.

Writing the file

  1. Edit the first group

    It starts as User-agent: * with /admin/, /checkout, /basket and /search disallowed. Add or remove paths, one per line.

  2. Add exceptions with Allow

    Put a more specific path in the Allow box to open part of a blocked folder, such as a help page inside /account/.

  3. Add groups for particular crawlers

    Add a crawler group creates a new block, starting as Googlebot; change the name to any user agent. Remove the last group takes one away.

  4. Decide on AI training crawlers

    Tick Block AI training crawlers to add whole-site blocks for GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot.

  5. List your sitemaps and download

    One address per line. They are written at the end of the file. Download robots.txt and upload it to the root of the site.

Example: a craft supplies shop with a private uploads folder

An online craft shop (fictional) keeps customer account pages out, except for its account help page, and does not want Google Images to pick up files in a private uploads folder. Two groups and one sitemap produce the file below. Allow lines are always written before Disallow lines within a group; order does not change the result, because the longest matching rule wins.

Input
Group 1: User agent *
  Disallow: /account/, /basket, /search
  Allow: /account/help
Group 2: User agent Googlebot-Image
  Disallow: /uploads/private/
Sitemap: https://www.pennycrafts.example/sitemap.xml
Result
User-agent: *
Allow: /account/help
Disallow: /account/
Disallow: /basket
Disallow: /search

User-agent: Googlebot-Image
Disallow: /uploads/private/

Sitemap: https://www.pennycrafts.example/sitemap.xml

Before uploading

  • No Disallow: / under User-agent: * unless you really mean to hide the whole site from search.
  • CSS, JavaScript and image folders are not blocked, since search engines need them to render your pages.
  • Any crawler with its own group repeats the general rules it should still follow, because it will ignore the * group.
  • The file opens at your domain followed by /robots.txt, in plain text.
  • You have tested a few key paths, such as the home page and a service page, with the robots.txt tester.

Questions people ask

Where does robots.txt go?

In the root of your domain, so that it opens at https://www.yourdomain/robots.txt. Each subdomain needs its own file. On WordPress, many SEO plugins manage a virtual robots.txt for you; on other hosts, upload the downloaded file through the file manager or FTP into the site's top folder.

What does an empty Disallow mean?

A group with Disallow: and nothing after it allows everything for that crawler. The generator writes it automatically when a group has no paths at all, so the group is still valid. It is a common way to say a particular crawler may visit the whole site.

Should I add a crawl delay?

Google's documentation says it does not support the Crawl-delay line, though Bing reads it, so it rarely helps a small site. This generator does not offer it. If a crawler is putting real strain on your server, your host's bot or rate limiting settings are a more reliable fix.

Does robots.txt keep private pages secret?

No. The file is public, so listing a private folder in it tells everyone where that folder is. Anything genuinely private should sit behind a login or be removed from the server. Use robots.txt to steer well-behaved crawlers, not to protect information.

Where your data goes

The rules you type are assembled into a file locally and offered as a download. No part of your site structure is uploaded or recorded.

Checked against the tool on

Top