NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Slate slate.com modified +18 −0
  • Blocked user-agent ‘ApifyBot’ entirely
  • Blocked user-agent ‘ApifyWebsiteContentCrawler’ entirely
  • Blocked user-agent ‘Bravebot’ entirely
  • Blocked user-agent ‘Brightbot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘Crawlspace’ entirely
  • … 12 more change(s)
view diff → snapshot
Qwant qwant.com modified +6 −2
  • Added Disallow: /?*bottomMenu=* for *
  • Added Disallow: /?*drawer=* for *
  • Added Disallow: /?*home=* for *
  • Removed Disallow: /account for *
  • 🤖 AI Added Disallow: /?*bottomMenu=* for AI2Bot
  • 🤖 AI Added Disallow: /?*drawer=* for AI2Bot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +48 −128
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 29 more change(s)
view diff → snapshot
Le Figaro lefigaro.fr modified +6 −0
  • 🤖 AI Added Allow: /horlogerie for ChatGPT-User
  • 🤖 AI Added Allow: /joaillerie for ChatGPT-User
  • 🤖 AI Added Allow: /mode-homme for ChatGPT-User
  • 🤖 AI Added Allow: /horlogerie for OAI-SearchBot
  • 🤖 AI Added Allow: /joaillerie for OAI-SearchBot
  • 🤖 AI Added Allow: /mode-homme for OAI-SearchBot
view diff → snapshot
IMDb imdb.com modified +1 −0
  • 🤖 AI Added rules for AI crawler DuckAssistBot (DuckDuckGo)
view diff → snapshot
Best Buy bestbuy.com modified +306 −76
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • Added rules for user-agent ‘Applebot’
  • Added rules for user-agent ‘BigSurAI’
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • … 26 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /resource-library/best-ai-app-for-logo-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-breadcrumb-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-design-changelog-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-skeleton-loader-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-split-test-variant-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/best-ai-app-for-logo-design/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −10
  • 🤖 AI Added Allow: /resource-library/ai-design-prompts/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/gallery/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /es-es/gallery/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /es-la/gallery/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /fr-fr/gallery/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /it-it/gallery/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Barracuda barracuda.com modified +27 −34
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for *
  • Removed Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for *
  • Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for DotBot
  • Removed Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for DotBot
  • Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for sistrix
  • … 3 more change(s)
view diff → snapshot
Uol uol.com.br modified +1 −1
  • 🤖 AI Added Allow: /*.ghtm$ for Google-Extended
  • 🤖 AI Removed Allow: /.ghtm$ for Google-Extended
view diff → snapshot
Figma Community figma.com modified +1 −1
  • 🤖 AI Added Allow: /webinars/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /gallery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/2026-ai-report/$ for Claude-SearchBot
  • 🤖 AI Removed Allow: /gallery/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/2026-ai-report/$ for Claude-User
  • 🤖 AI Removed Allow: /gallery/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/sightlines-issue-no2/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/sightlines-issue-no2/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/sightlines-issue-no2/$ for Claude-User
  • 🤖 AI Added Allow: /blog/sightlines-issue-no2/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/sightlines-issue-no2/$ for PerplexityBot
view diff → snapshot
The Economist economist.com modified +16 −0
  • 🤖 AI Added Allow: /syndication$ for ChatGPT-User
  • 🤖 AI Added Allow: /syndication/ for ChatGPT-User
  • 🤖 AI Added Allow: /syndication$ for Claude-SearchBot
  • 🤖 AI Added Allow: /syndication/ for Claude-SearchBot
  • 🤖 AI Added Allow: /syndication$ for Claude-User
  • 🤖 AI Added Allow: /syndication/ for Claude-User
  • … 10 more change(s)
view diff → snapshot
Vox vox.com modified +100 −0
  • Added Allow: /strategist for A6-Indexer
  • Added Allow: /strategist for AhrefsBot
  • 🤖 AI Added Allow: /strategist for AI2Bot
  • Added Allow: /strategist for AI2Bot-DeepResearchEval
  • Added Allow: /strategist for Ai2Bot-Dolma
  • Added Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
Samsung Electronics samsung.com modified +8 −3
  • Added Allow: /ca/multistore/offers/back-to-school/ for *
  • Added Allow: /ca_fr/multistore/offers/back-to-school/ for *
  • Added Allow: /ca/multistore/offers/back-to-school/ for AdsBot-Google
  • Added Allow: /ca_fr/multistore/offers/back-to-school/ for AdsBot-Google
  • 🤖 AI Added Allow: /ca/multistore/offers/back-to-school/ for ChatGPT-User
  • 🤖 AI Added Allow: /ca_fr/multistore/offers/back-to-school/ for ChatGPT-User
  • … 17 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Vox vox.com modified +0 −100
  • Removed Allow: /strategist for A6-Indexer
  • Removed Allow: /strategist for AhrefsBot
  • 🤖 AI Removed Allow: /strategist for AI2Bot
  • Removed Allow: /strategist for AI2Bot-DeepResearchEval
  • Removed Allow: /strategist for Ai2Bot-Dolma
  • Removed Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /uk-tax-strategy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /uk-tax-strategy/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for Claude-User
  • 🤖 AI Added Allow: /uk-tax-strategy/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Vox vox.com modified +100 −0
  • Added Allow: /strategist for A6-Indexer
  • Added Allow: /strategist for AhrefsBot
  • 🤖 AI Added Allow: /strategist for AI2Bot
  • Added Allow: /strategist for AI2Bot-DeepResearchEval
  • Added Allow: /strategist for Ai2Bot-Dolma
  • Added Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Reliance Steel & Aluminum rsac.com modified +23 −7
  • Blocked user-agent ‘AhrefsBot’ entirely
  • Blocked user-agent ‘BLEXBot’ entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • … 15 more change(s)
view diff → snapshot
Renault Group renaultgroup.com modified +0 −72
  • 🤖 AI Unblocked ClaudeBot (Anthropic)
  • 🤖 AI Removed Allow: /br/ for ClaudeBot
  • 🤖 AI Removed Allow: /de/ for ClaudeBot
  • 🤖 AI Removed Allow: /es/ for ClaudeBot
  • 🤖 AI Removed Allow: /it/ for ClaudeBot
  • 🤖 AI Removed Allow: /pt/ for ClaudeBot
  • … 8 more change(s)
view diff → snapshot
DOAJ doaj.org modified +20 −68
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 21 more change(s)
view diff → snapshot
Indeed indeed.com modified +30 −0
  • Added Allow: /*&start=0& for *
  • Added Allow: /*&start=10& for *
  • Added Allow: /*&start=20& for *
  • Added Allow: /*&start=30& for *
  • Added Allow: /*&start=40& for *
  • Added Allow: /*&start=50& for *
  • … 45 more change(s)
view diff → snapshot
AstraZeneca astrazeneca.com modified +68 −5
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • Added rules for user-agent ‘AndiBot’
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 17 more change(s)
view diff → snapshot
Itaú Unibanco itau.com.br modified +112 −29
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘BingBot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • Added rules for user-agent ‘facebookexternalhit’
  • … 43 more change(s)
view diff → snapshot
Peru (Gob.pe) gob.pe modified +14 −0
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • Added Disallow: /*&sheet= for *
  • Added Disallow: /*?sheet= for *
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for Claude-User
  • 🤖 AI Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for PerplexityBot
view diff → snapshot