AI Crawler Checker
Check whether ChatGPT, Claude, Perplexity, Google and other AI crawlers can read your website. Enter an address to see what your robots.txt tells each AI bot, the exact line that decides it, and whether you have an llms.txt file.
- 17 AI crawlers
- Search and training split
- Line by line reasons
- No signup
AI search and answers
These decide whether your pages can be found, quoted and linked in AI answers.
OAI-SearchBot Claude-SearchBot PerplexityBot meta-webindexer Googlebot Bingbot Applebot
Fetches on request
These visit a page only when a person asks an assistant about it.
ChatGPT-User Claude-User Perplexity-User meta-externalfetcher
Model training
These collect pages to train AI models. Blocking them does not stop you appearing in AI search.
GPTBot ClaudeBot Google-Extended Applebot-Extended meta-externalagent CCBot
Build your AI rules
# AI search and answer engines User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: meta-webindexer User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: meta-externalfetcher Allow: / # AI model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot Disallow: /
The most common choice for sites that want visitors from AI search. Your pages can still be quoted and linked in ChatGPT, Claude, Perplexity and Meta AI answers.
One thing to know before you paste: a bot named in its own group ignores your group for all bots. If your User-agent: * group blocks private folders, copy those Disallow lines into the AI groups too.
Run a full website auditWhich AI Crawlers Should You Allow?
AI crawlers do three different jobs, and robots.txt lets you treat each one differently. Search crawlers decide whether your pages can be found and cited in AI answers. Fetchers visit a page when someone asks an assistant about it. Training crawlers collect pages to build AI models.
The most common mistake is blocking a training crawler and thinking it removes you from AI search, or blocking a search crawler by accident while trying to stop training. Blocking GPTBot does not take you out of ChatGPT search, because ChatGPT search uses OAI-SearchBot. Blocking Google-Extended does not affect Google Search, because Googlebot does the crawling.
AI search and answers
These decide whether your pages can be found, quoted and linked in AI answers.
Fetches on request
These visit a page only when a person asks an assistant about it.
Model training
These collect pages to train AI models. Blocking them does not stop you appearing in AI search.
AI Crawler User Agents and What They Do
The robots.txt name for each AI crawler, taken from each company's own documentation. Checked November 6, 2026.
| robots.txt name | Company | Used for | Follows robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show and link in ChatGPT search results. | Yes |
| Claude-SearchBot | Anthropic | Indexes pages to improve the answers Claude gives when it searches the web. | Yes |
| PerplexityBot | Perplexity | Builds the index Perplexity uses to show and link pages in its answers. | Yes |
| meta-webindexer | Meta | Indexes pages to improve Meta AI search results. | Yes |
| Googlebot | Crawls for Google Search, including its AI features. Blocking it takes you out of Google. | Yes | |
| Bingbot | Microsoft | Crawls for Bing search, which Microsoft Copilot also draws on. | Yes |
| Applebot | Apple | Powers Apple search features. Its data may also train Apple models unless Applebot-Extended is blocked. | Yes |
| ChatGPT-User | OpenAI | Visits a page when someone asks ChatGPT or a custom GPT about it. | May ignore it |
| Claude-User | Anthropic | Visits a page when someone asks Claude a question that needs it. | Yes |
| Perplexity-User | Perplexity | Fetches a page when a question in Perplexity needs it. | May ignore it |
| meta-externalfetcher | Meta | Fetches a link a person asks Meta AI about. | May ignore it |
| GPTBot | OpenAI | Collects pages that may be used to train OpenAI models. | Yes |
| ClaudeBot | Anthropic | Collects pages that may be used to train Anthropic models. | Yes |
| Google-Extended | Not a crawler. Tells Google whether pages Googlebot crawls may train Gemini and ground its answers. Google says it does not affect Google Search. | Not a crawler | |
| Applebot-Extended | Apple | Not a crawler. Tells Apple whether pages Applebot crawls may train its models. Search features are not affected. | Not a crawler |
| meta-externalagent | Meta | Collects pages to train Meta AI models and improve its products. | Yes |
| CCBot | Common Crawl | Builds a free public archive of the web that many AI models are trained on. | Yes |
How to Allow AI Search but Block AI Training
Put each kind of crawler in its own group. One group can list several user agents, and a blank line between groups keeps them apart:
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: /
A crawler follows only the group that names it, and ignores your group for all bots (User-agent: *). So if that group blocks folders such as /admin/ or /cart/, repeat those lines inside the AI groups. The rule builder in the checker above gives you the full set to copy.
How the Checker Reads robots.txt
It follows the robots.txt standard (RFC 9309) the way Google applies it, so the answer matches what a well behaved crawler would do:
- The most specific group wins. A bot uses the group that names it. Only if none does, it uses the * group.
- The longest matching rule wins. Allow: /blog/ beats Disallow: / for a blog page, and Allow wins a tie.
- No file means no limits. A robots.txt that returns 404 lets every crawler in.
- A broken file means wait. Google treats a server error on robots.txt as a temporary block on the whole site, and other crawlers tend to do the same.
- A web page is not a robots.txt. Some sites answer /robots.txt with their homepage. Crawlers find no rules in that, so everything is allowed.
What robots.txt Cannot Do
- It does not block anything by force. It asks politely. Firewalls and CDN bot settings, such as Cloudflare's, can block AI crawlers even when robots.txt allows them, and this checker cannot see those from outside.
- It does not remove what was already collected. A new Disallow only affects future crawling.
- Some fetchers may skip it. OpenAI, Perplexity and Meta say their fetchers that act for a person may not follow robots.txt.
To see how AI crawler access fits with the rest of your technical setup, run the free website audit, which includes this check alongside more than 70 others.
Frequently Asked Questions About AI Crawlers
Does blocking GPTBot remove my site from ChatGPT?
Does blocking Google-Extended hurt my Google rankings?
Which AI crawlers should I allow?
Can robots.txt stop every AI bot?
How long do robots.txt changes take to work?
What is a Content-Signal line in robots.txt?
Do I need an llms.txt file?
Is the AI crawler checker free?
Related SEO Tools and Guides
Official crawler pages
Bot list checked against these pages on November 6, 2026.