AI crawlers and robots.txt: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot
To appear in AI answers, allow the search crawlers—OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity and Googlebot for Google’s AI features—in robots.txt. Training crawlers such as GPTBot and ClaudeBot are separate and can be allowed or blocked independently.
Which AI crawlers exist and what do they do?
| User agent | Operator | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search answers |
| GPTBot | OpenAI | Crawls content that may be used to train foundation models |
| ChatGPT-User | OpenAI | Fetches pages for user-initiated actions in ChatGPT |
| Claude-SearchBot | Anthropic | Improves search result quality for Claude |
| ClaudeBot | Anthropic | Collects data for model training |
| Claude-User | Anthropic | Fetches pages when a Claude user asks |
| PerplexityBot | Perplexity | Indexes pages for Perplexity search results |
| Perplexity-User | Perplexity | Fetches pages in response to a user question |
| Googlebot | Crawls for Search, including AI Overviews and AI Mode | |
| Google-Extended | Controls use of content for Gemini training and grounding outside Search |
What robots.txt settings maximize AI visibility?
Each crawler is controlled independently. A site that wants maximum visibility allows the search and user crawlers; whether to allow training crawlers is a separate business decision. OpenAI notes that changes to robots.txt can take about 24 hours to take effect for OAI-SearchBot.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
# Training crawlers: allow or disallow independently
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /Why do AI crawlers still fail to read a site?
- CDN or firewall rules that block non-browser user agents, even when robots.txt allows them.
- Content that only appears after JavaScript runs—many crawlers read the initial HTML only.
- Slow responses, redirect chains and error pages on important URLs.
- Blocking search crawlers by accident while trying to block training crawlers.
Do you need an llms.txt file?
It is optional. Google states that no new machine-readable or AI text files are needed to appear in its AI features, and an Ahrefs analysis reported that 97% of llms.txt files received zero requests in May 2026. A short, factual llms.txt does no harm, but robots.txt, sitemaps and clean HTML matter far more.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI controls each crawler independently: blocking GPTBot opts out of training, while OAI-SearchBot determines whether a site can appear in ChatGPT search answers.
How do I get new pages discovered faster?
Keep an up-to-date XML sitemap, submit it in Google Search Console and Bing Webmaster Tools, and notify participating search engines of changed URLs through IndexNow.