AI & search crawlers
Every crawler NeuralCrawl tracks in robots.txt — vendor, purpose, and how many monitored sites block it by name.
AI crawlers 28 bots
| Bot | Vendor | Purpose | Sites blocking |
|---|---|---|---|
GPTBot |
OpenAI | LLM training crawler | 111 (9.3%) |
CCBot |
Common Crawl | Open web corpus (used for LLM training) | 110 (9.2%) |
ClaudeBot |
Anthropic | LLM training crawler | 103 (8.6%) |
Bytespider |
ByteDance | LLM training crawler | 101 (8.4%) |
Google-Extended |
Gemini training opt-out token | 94 (7.9%) | |
meta-externalagent |
Meta | Meta AI training crawler | 91 (7.6%) |
Applebot-Extended |
Apple | Apple Intelligence training opt-out | 84 (7.0%) |
Amazonbot |
Amazon | Alexa / LLM crawler | 75 (6.3%) |
anthropic-ai |
Anthropic | Legacy crawler token | 68 (5.7%) |
Diffbot |
Diffbot | Structured data extraction | 66 (5.5%) |
omgili |
Webz.io | Data feeds resold for AI training | 64 (5.4%) |
PerplexityBot |
Perplexity | Answer-engine indexing | 62 (5.2%) |
FacebookBot |
Meta | Meta AI crawler (legacy) | 58 (4.8%) |
cohere-ai |
Cohere | LLM training crawler | 57 (4.8%) |
Claude-Web |
Anthropic | Legacy crawler token | 52 (4.3%) |
ChatGPT-User |
OpenAI | User-triggered browsing | 50 (4.2%) |
Timpibot |
Timpi | Decentralised index crawler | 42 (3.5%) |
Claude-User |
Anthropic | User-triggered browsing | 39 (3.3%) |
Claude-SearchBot |
Anthropic | Search indexing | 36 (3.0%) |
YouBot |
You.com | Answer-engine indexing | 36 (3.0%) |
Perplexity-User |
Perplexity | User-triggered browsing | 35 (2.9%) |
AI2Bot |
Allen Institute for AI | Research crawler | 34 (2.8%) |
meta-externalfetcher |
Meta | User-triggered fetcher | 33 (2.8%) |
OAI-SearchBot |
OpenAI | Search indexing | 31 (2.6%) |
DuckAssistBot |
DuckDuckGo | DuckAssist answers | 30 (2.5%) |
PanguBot |
Huawei | LLM training crawler | 27 (2.3%) |
MistralAI-User |
Mistral AI | User-triggered browsing | 26 (2.2%) |
Meta-WebIndexer |
Meta | Meta AI search indexing & citations | 13 (1.1%) |
Search-engine crawlers 12 bots
| Bot | Vendor | Purpose | Sites blocking |
|---|---|---|---|
PetalBot |
Huawei | Petal Search index | 55 (4.6%) |
Baiduspider |
Baidu | Baidu Search index | 28 (2.3%) |
Applebot |
Apple | Siri & Spotlight Search index | 10 (0.8%) |
YandexBot |
Yandex | Yandex Search index | 8 (0.7%) |
SeznamBot |
Seznam | Seznam Search index (Czechia) | 7 (0.6%) |
Slurp |
Yahoo | Yahoo Search index | 4 (0.3%) |
Sogou |
Sogou | Sogou Search index (China) | 4 (0.3%) |
Googlebot-News |
Google News index | 2 (0.2%) | |
Bingbot |
Microsoft | Bing Search index | 0 (0.0%) |
DuckDuckBot |
DuckDuckGo | DuckDuckGo Search index | 0 (0.0%) |
Googlebot |
Google Search index | 0 (0.0%) | |
Googlebot-Image |
Google Images index | 0 (0.0%) |
See also The Wall — a site-by-site matrix of blocking status across the most newsworthy crawlers.