# AI Crawler robots.txt Generator

> Allow or block GPTBot, ClaudeBot, Google-Extended, PerplexityBot and 14 other AI crawlers. Block training but stay in AI search. Copy or download robots.txt. Free.

Canonical: https://promptitin.com/free-tools/ai-crawler-robots-txt-generator
Last updated: 2026-11-13

Decide which AI crawlers can read your site. Block training, stay visible in ChatGPT and Perplexity search, and get a robots.txt you can paste in.

This is an interactive tool. Use it at https://promptitin.com/free-tools/ai-crawler-robots-txt-generator.

## How to use it

1. Start from a preset: allow all, block training only, or block all AI.
2. Switch any crawler between Allowed and Blocked.
3. Add paths every bot should stay out of, and your sitemap URL.
4. Copy or download robots.txt and put it at the root of your site.

## Tips

- Blocking training crawlers doesn't remove you from Google Search or AI search results.
- Blocking search crawlers like OAI-SearchBot can stop AI search from showing and linking your pages.
- A bot named in its own group ignores the * group, so private paths are repeated in the allowed group. The tool does this for you.
- robots.txt is a request, not a lock. Use logins or a firewall for content that must stay private.

## FAQ

### Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and links people open in a chat use ChatGPT-User. Block only GPTBot to opt out of training and stay in ChatGPT search.

### Does Google-Extended affect Google Search?

No. It only controls whether Google may use your pages to train and ground Gemini. Your Google Search ranking isn't affected.

### Do all AI bots follow robots.txt?

The big AI companies say their crawlers do. Fetchers that act for a user, like Perplexity-User, may not, because a person asked for the page. Some lesser-known scrapers ignore it.

### Where does robots.txt go?

At the root of your domain, so it loads at https://yoursite.com/robots.txt. If you already have one, merge these groups into it.

## Related

- [llms.txt Generator](https://promptitin.com/free-tools/llms-txt-generator)
- [Sitemap Checker](https://promptitin.com/free-tools/sitemap-checker)
- [Sitemap URL Extractor](https://promptitin.com/free-tools/sitemap-url-extractor)