PromptItIn logoPromptItIn
Log in
SEO & GEO

AI crawler robots.txt generator

Decide which AI crawlers can read your site. Block training, stay visible in ChatGPT and Perplexity search, and get a robots.txt you can paste in.

Free tool · Updated

Start from
Then switch any crawler below.
Training crawlersCollect pages to train AI models. Blocking them doesn't affect Google or Bing search.

GPTBot

OpenAI

ClaudeBot

Anthropic

Google-Extended

Google (Gemini)

Applebot-Extended

Apple

meta-externalagent

Meta

CCBot

Common Crawl

Bytespider

ByteDance

Amazonbot

Amazon

cohere-ai

Cohere

AI search crawlersIndex pages so AI search can show, cite and link to them.

OAI-SearchBot

OpenAI (ChatGPT search)

Claude-SearchBot

Anthropic

PerplexityBot

Perplexity

DuckAssistBot

DuckDuckGo

YouBot

You.com

Fetchers for usersOpen a page when someone asks an AI about it.

ChatGPT-User

OpenAI

Claude-User

Anthropic

Perplexity-User

Perplexity

MistralAI-User

Mistral

robots.txt

User-agent: * Allow: / # AI crawlers you allow User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: DuckAssistBot User-agent: YouBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: MistralAI-User Allow: / # AI crawlers you block User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: cohere-ai Disallow: /

How to use it

  1. Start from a preset: allow all, block training only, or block all AI.
  2. Switch any crawler between Allowed and Blocked.
  3. Add paths every bot should stay out of, and your sitemap URL.
  4. Copy or download robots.txt and put it at the root of your site.

Tips

  • Blocking training crawlers doesn't remove you from Google Search or AI search results.
  • Blocking search crawlers like OAI-SearchBot can stop AI search from showing and linking your pages.
  • A bot named in its own group ignores the * group, so private paths are repeated in the allowed group. The tool does this for you.
  • robots.txt is a request, not a lock. Use logins or a firewall for content that must stay private.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and links people open in a chat use ChatGPT-User. Block only GPTBot to opt out of training and stay in ChatGPT search.

Do all AI bots follow robots.txt?

The big AI companies say their crawlers do. Fetchers that act for a user, like Perplexity-User, may not, because a person asked for the page. Some lesser-known scrapers ignore it.

Where does robots.txt go?

At the root of your domain, so it loads at https://yoursite.com/robots.txt. If you already have one, merge these groups into it.

llms.txt GeneratorCreate an llms.txt file in the standard format: your site's name, a summary and your key pages with notes. Paste links in any form. Free, runs in your browser.Sitemap CheckerPaste your sitemap.xml to check it against the sitemap rules: bad URLs, duplicates, wrong dates, mixed hosts, http links and size limits. Free, runs in your browser.Sitemap URL ExtractorPaste a sitemap.xml and pull out every URL as a clean list, paths only, or a CSV with lastmod dates. Filter by folder and download. Free, runs in your browser.

All free tools

Want a full prompt, not just keywords?

PromptItIn turns a rough idea into a clear prompt for ChatGPT, Claude, Gemini or any image model. Free to start.

Open the prompt generatorAdd to Chrome, free