The four kinds of AI user agent
The tool groups 26 published user-agent tokens by what they do. Blocking one kind does not block the others.
| Group | Examples in the tool | Blocking it means |
|---|---|---|
| Collect pages to train AI models | GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot | Asking that your pages are not gathered for model training |
| Opt-out tokens read by existing search crawlers | Google-Extended, Applebot-Extended | Google and Apple keep crawling for search, but are asked not to use your pages for AI training |
| Index pages for AI search answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot | Your pages may not appear or be cited in those AI search products |
| Fetch a page when a person asks an AI assistant | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User | An assistant cannot open your page when someone asks it to |
Merging the blocks
Paste your current file
Open your domain followed by
/robots.txt, copy everything and paste it. Leave the box empty to start a new file.Pick the crawlers
Block AI training only ticks the training and opt-out groups. Block every AI crawler listed ticks all four. You can then tick or untick individual tokens, and type any others in the extra box.
Choose the layout
By default each crawler gets its own group. Tick Write one group with several User-agent lines for a shorter file; RFC 9309 allows both.
Read the figures and warnings
Groups added, Already in your file and Blocked after merge show what happened. A warning appears if a crawler already had its own group, or if your file already blocks everything.
Download and upload
Save
robots.txtand replace the file at the root of your site.
Example: a shop that had already allowed GPTBot
An online ironmonger (fictional) once added a rule allowing GPTBot. It now wants four AI tokens blocked. Three are added, but GPTBot already has its own group, so the tool leaves it alone and says so. Blocked after merge reads 3 of 4: the existing Allow: / still applies to GPTBot until you edit that group yourself.
User-agent: * Disallow: /basket Disallow: /account/ User-agent: GPTBot Allow: / Sitemap: https://www.oakandiron.example/sitemap.xml Chosen: GPTBot, ClaudeBot, CCBot, Google-Extended
User-agent: * Disallow: /basket Disallow: /account/ User-agent: GPTBot Allow: / # AI crawlers: blocked from the whole site User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://www.oakandiron.example/sitemap.xml GPTBot already has its own group in your file, so it was left unchanged.
Questions people ask
Will blocking AI crawlers hurt my Google ranking?
Blocking Google-Extended does not stop Googlebot, so ordinary search crawling continues; Google documents it as a control over AI model use only. Blocking AI search bots such as OAI-SearchBot or PerplexityBot may keep you out of those products' answers, which is a separate trade-off for each business to weigh.
Why was a crawler I chose not added?
Under the robots.txt standard a crawler follows only the groups that name it, so adding a second group for the same token would be confusing. If your file already has a group for that crawler, the tool leaves it as it is and warns you. Edit that group by hand to change what it allows.
Should a small business block AI crawlers at all?
It depends on what you want. If people finding you through AI assistants matters, blocking the search and user groups works against you. If your concern is your text and photos being used for training, blocking the training group and the opt-out tokens is the narrower choice. The tool lets you pick group by group.
Can I block AI crawlers from only part of my site?
Yes in principle, by giving those crawlers Disallow lines for specific folders instead of the whole site. This tool writes whole-site blocks, which is what most people want. For partial rules, use the robots.txt generator and add a group per crawler with the paths you choose.
Where your data goes
Your existing robots.txt is merged and tested by script in the browser. The file is not fetched from your domain and the result is not sent anywhere.
Checked against the tool on