AI & search crawlers
Every crawler NeuralCrawl tracks in robots.txt — vendor, purpose, and how many monitored sites block it by name.
AI crawlers 28 bots
| Bot | Vendor | Purpose | Sites blocking |
|---|---|---|---|
GPTBot |
OpenAI | LLM training crawler | 112 (9.3%) |
CCBot |
Common Crawl | Open web corpus (used for LLM training) | 110 (9.1%) |
Bytespider |
ByteDance | LLM training crawler | 105 (8.7%) |
ClaudeBot |
Anthropic | LLM training crawler | 105 (8.7%) |
Google-Extended |
Gemini training opt-out token | 94 (7.8%) | |
meta-externalagent |
Meta | Meta AI training crawler | 91 (7.6%) |
Applebot-Extended |
Apple | Apple Intelligence training opt-out | 89 (7.4%) |
Diffbot |
Diffbot | Structured data extraction | 78 (6.5%) |
anthropic-ai |
Anthropic | Legacy crawler token | 76 (6.3%) |
Amazonbot |
Amazon | Alexa / LLM crawler | 75 (6.2%) |
omgili |
Webz.io | Data feeds resold for AI training | 74 (6.2%) |
cohere-ai |
Cohere | LLM training crawler | 66 (5.5%) |
PerplexityBot |
Perplexity | Answer-engine indexing | 63 (5.2%) |
FacebookBot |
Meta | Meta AI crawler (legacy) | 61 (5.1%) |
Claude-Web |
Anthropic | Legacy crawler token | 60 (5.0%) |
ChatGPT-User |
OpenAI | User-triggered browsing | 51 (4.2%) |
Timpibot |
Timpi | Decentralised index crawler | 43 (3.6%) |
Claude-SearchBot |
Anthropic | Search indexing | 41 (3.4%) |
Claude-User |
Anthropic | User-triggered browsing | 41 (3.4%) |
YouBot |
You.com | Answer-engine indexing | 39 (3.2%) |
Perplexity-User |
Perplexity | User-triggered browsing | 38 (3.2%) |
AI2Bot |
Allen Institute for AI | Research crawler | 37 (3.1%) |
meta-externalfetcher |
Meta | User-triggered fetcher | 34 (2.8%) |
OAI-SearchBot |
OpenAI | Search indexing | 33 (2.7%) |
DuckAssistBot |
DuckDuckGo | DuckAssist answers | 31 (2.6%) |
PanguBot |
Huawei | LLM training crawler | 29 (2.4%) |
MistralAI-User |
Mistral AI | User-triggered browsing | 27 (2.2%) |
Meta-WebIndexer |
Meta | Meta AI search indexing & citations | 16 (1.3%) |
Search-engine crawlers 12 bots
| Bot | Vendor | Purpose | Sites blocking |
|---|---|---|---|
PetalBot |
Huawei | Petal Search index | 64 (5.3%) |
Baiduspider |
Baidu | Baidu Search index | 36 (3.0%) |
YandexBot |
Yandex | Yandex Search index | 9 (0.7%) |
Applebot |
Apple | Siri & Spotlight Search index | 8 (0.7%) |
SeznamBot |
Seznam | Seznam Search index (Czechia) | 8 (0.7%) |
Slurp |
Yahoo | Yahoo Search index | 4 (0.3%) |
Sogou |
Sogou | Sogou Search index (China) | 4 (0.3%) |
Googlebot-News |
Google News index | 2 (0.2%) | |
Bingbot |
Microsoft | Bing Search index | 1 (0.1%) |
DuckDuckBot |
DuckDuckGo | DuckDuckGo Search index | 0 (0.0%) |
Googlebot |
Google Search index | 0 (0.0%) | |
Googlebot-Image |
Google Images index | 0 (0.0%) |
See also The Wall — a site-by-site matrix of blocking status across the most newsworthy crawlers.