Map Extract
Free tool

Block AI crawlers in your robots.txt

Start from the robots.txt you already publish, choose which AI user agents should stay out, and the tool writes a merged version. Existing rules are kept, sitemap lines move to the end, and every chosen token is tested against the result.

Runs in your browser. Nothing you enter is uploaded.

The four kinds of AI user agent

The tool groups 26 published user-agent tokens by what they do. Blocking one kind does not block the others.

GroupExamples in the toolBlocking it means
Collect pages to train AI modelsGPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, AmazonbotAsking that your pages are not gathered for model training
Opt-out tokens read by existing search crawlersGoogle-Extended, Applebot-ExtendedGoogle and Apple keep crawling for search, but are asked not to use your pages for AI training
Index pages for AI search answersOAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBotYour pages may not appear or be cited in those AI search products
Fetch a page when a person asks an AI assistantChatGPT-User, Claude-User, Perplexity-User, MistralAI-UserAn assistant cannot open your page when someone asks it to

Merging the blocks

  1. Paste your current file

    Open your domain followed by /robots.txt, copy everything and paste it. Leave the box empty to start a new file.

  2. Pick the crawlers

    Block AI training only ticks the training and opt-out groups. Block every AI crawler listed ticks all four. You can then tick or untick individual tokens, and type any others in the extra box.

  3. Choose the layout

    By default each crawler gets its own group. Tick Write one group with several User-agent lines for a shorter file; RFC 9309 allows both.

  4. Read the figures and warnings

    Groups added, Already in your file and Blocked after merge show what happened. A warning appears if a crawler already had its own group, or if your file already blocks everything.

  5. Download and upload

    Save robots.txt and replace the file at the root of your site.

Example: a shop that had already allowed GPTBot

An online ironmonger (fictional) once added a rule allowing GPTBot. It now wants four AI tokens blocked. Three are added, but GPTBot already has its own group, so the tool leaves it alone and says so. Blocked after merge reads 3 of 4: the existing Allow: / still applies to GPTBot until you edit that group yourself.

Input
User-agent: *
Disallow: /basket
Disallow: /account/

User-agent: GPTBot
Allow: /

Sitemap: https://www.oakandiron.example/sitemap.xml

Chosen: GPTBot, ClaudeBot, CCBot, Google-Extended
Result
User-agent: *
Disallow: /basket
Disallow: /account/

User-agent: GPTBot
Allow: /

# AI crawlers: blocked from the whole site
User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.oakandiron.example/sitemap.xml

GPTBot already has its own group in your file, so it was left unchanged.

Questions people ask

Will blocking AI crawlers hurt my Google ranking?

Blocking Google-Extended does not stop Googlebot, so ordinary search crawling continues; Google documents it as a control over AI model use only. Blocking AI search bots such as OAI-SearchBot or PerplexityBot may keep you out of those products' answers, which is a separate trade-off for each business to weigh.

Why was a crawler I chose not added?

Under the robots.txt standard a crawler follows only the groups that name it, so adding a second group for the same token would be confusing. If your file already has a group for that crawler, the tool leaves it as it is and warns you. Edit that group by hand to change what it allows.

Should a small business block AI crawlers at all?

It depends on what you want. If people finding you through AI assistants matters, blocking the search and user groups works against you. If your concern is your text and photos being used for training, blocking the training group and the opt-out tokens is the narrower choice. The tool lets you pick group by group.

Can I block AI crawlers from only part of my site?

Yes in principle, by giving those crawlers Disallow lines for specific folders instead of the whole site. This tool writes whole-site blocks, which is what most people want. For partial rules, use the robots.txt generator and add a group per crawler with the paths you choose.

Where your data goes

Your existing robots.txt is merged and tested by script in the browser. The file is not fetched from your domain and the result is not sent anywhere.

Checked against the tool on

Top