NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Hiding initial archives
GitHub github.com modified +18 −1
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • Added Allow: /enterprise/contact/data-residency?* for *
  • … 34 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +11 −0
  • 🤖 AI Added Allow: /de-de/legal/privacy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/legal/tos/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/legal/privacy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/legal/tos/$ for CHATGPT-User
  • 🤖 AI Added Allow: /hi-in/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/legal/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Delta Air Lines delta.com modified +0 −58
  • Removed all rules for user-agent ‘ai2-crawler’
  • 🤖 AI Removed rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Removed rules for AI crawler CCBot (Common Crawl)
  • … 13 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +68 −1
  • 🤖 AI Added Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
  • 🤖 AI Added Allow: /bespoke-content/ for AI2Bot
  • 🤖 AI Added Allow: /betting/ for AI2Bot
  • 🤖 AI Added Allow: /brand-usa for AI2Bot
  • 🤖 AI Added Allow: /branded-content/ for AI2Bot
  • 🤖 AI Added Allow: /british-airways-holidays/cyprus-hotel-map for AI2Bot
  • … 45 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +0 −14
  • Removed Disallow: /4624/TheAtlanticOnline/* for *
  • Removed Disallow: /4624/TheAtlanticOnline/* for AmazonAdBot
  • Removed Disallow: /4624/TheAtlanticOnline/* for Applebot
  • Removed Disallow: /4624/TheAtlanticOnline/* for ArcSpanScraper
  • Removed Disallow: /4624/TheAtlanticOnline/* for BingBot
  • 🤖 AI Removed Disallow: /4624/TheAtlanticOnline/* for ChatGPT-User
  • … 8 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +11 −17
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked Claude-User (Anthropic) entirely
  • … 4 more change(s)
view diff → snapshot
Le Monde lemonde.fr modified +137 −3
  • Blocked user-agent ‘AgentDataBot’ entirely
  • Blocked user-agent ‘AGI Agent’ entirely
  • Blocked user-agent ‘AIWebIndex-Agent’ entirely
  • Blocked user-agent ‘AndiAgent’ entirely
  • Blocked user-agent ‘artsdata-crawler’ entirely
  • Blocked user-agent ‘BeansLLM-CorpusBot’ entirely
  • … 42 more change(s)
view diff → snapshot
First American Financial firstam.com modified +129 −5
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 32 more change(s)
view diff → snapshot
Flipkart flipkart.com modified +8 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Amzn-SearchBot’ entirely
  • Added Disallow: */all/* for *
  • Added Disallow: */filters for *
view diff → snapshot
Best Buy bestbuy.com modified +14 −4
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Brightbot’
  • 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
  • 🤖 AI Added rules for AI crawler DuckAssistBot (DuckDuckGo)
  • Added rules for user-agent ‘Gemini-AI’
  • Added rules for user-agent ‘GoogleAgent-Mariner’
  • … 36 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /ja-jp/partners/find-a-service-partner/sun-asterisk/$ for CHATGPT-User
  • 🤖 AI Added Allow: /partners/find-a-service-partner/sun-asterisk/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/partners/find-a-service-partner/sun-asterisk/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /partners/find-a-service-partner/sun-asterisk/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ja-jp/partners/find-a-service-partner/sun-asterisk/$ for Claude-User
  • 🤖 AI Added Allow: /partners/find-a-service-partner/sun-asterisk/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-build-with-code-on-the-canvas-using-the-figma-agent-and-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/how-figma-manages-sandboxes-at-scale/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/brazil-general-data-protection-law/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/how-figma-manages-sandboxes-at-scale/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/brazil-general-data-protection-law/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-manages-sandboxes-at-scale/$ for Claude-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/brazil-general-data-protection-law/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +160 −2
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘Ai2Bot-Dolma’ entirely
  • Blocked user-agent ‘Amazon Kendra’ entirely
  • Blocked user-agent ‘Arquivo-web-crawler’ entirely
  • Blocked user-agent ‘Barkrowler’ entirely
  • Blocked user-agent ‘Big Sur AI’ entirely
  • … 45 more change(s)
view diff → snapshot
Samsung Electronics samsung.com modified +2 −1
  • Added Allow: /sg/multistore/sg_eppfinancing/offers_epp/ for *
  • Removed Allow: /sg/multistore/sg_eppfinancing/offers_epp/sc_premium/ for *
  • Added Allow: /sg/multistore/sg_eppfinancing/offers_epp/ for AdsBot-Google
  • Removed Allow: /sg/multistore/sg_eppfinancing/offers_epp/sc_premium/ for AdsBot-Google
  • 🤖 AI Added Allow: /sg/multistore/sg_eppfinancing/offers_epp/ for ChatGPT-User
  • 🤖 AI Removed Allow: /sg/multistore/sg_eppfinancing/offers_epp/sc_premium/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −3
  • 🤖 AI Added Allow: /es-es/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /nl-nl/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/legal/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Bayer bayer.com modified +79 −0
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 9 more change(s)
view diff → snapshot
Varonis varonis.com modified +10 −5
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • Removed Disallow: /*?*hsCacheBuster=* for *
  • Removed Disallow: /*?*hs_preview=* for *
  • Removed Disallow: /_hcms/preview/ for *
  • Removed Disallow: /hs/manage-preferences/ for *
  • … 3 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-ben-ganz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-ben-ganz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-ben-ganz/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +3 −70
  • Added Allow: / for *
  • Added Disallow: /promotions/emails/ for *
  • Added Disallow: /sponsored/travel/msc-cruises/ for *
  • 🤖 AI Removed Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
  • 🤖 AI Removed Allow: /bespoke-content/ for AI2Bot
  • 🤖 AI Removed Allow: /betting/ for AI2Bot
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Bloomberg bloomberg.com modified +43 −255
  • 🤖 AI Added Allow: /?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa/?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa/llms.txt/?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa/llms.txt?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /asia/?bloomberg_debug= for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +11 −0
  • 🤖 AI Added Allow: /blog/figma-and-sanrio-celebrate-hello-kitty-and-friends/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/6-ways-teams-are-experimenting-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/6-ways-teams-are-experimenting-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/6-ways-teams-are-experimenting-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/6-ways-teams-are-experimenting-with-code-layers/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
The Hill thehill.com modified +1 −2
  • 🤖 AI Unblocked Google-Extended (Google)
  • 🤖 AI Added Allow: / for Google-Extended
  • 🤖 AI Removed Allow: /advertise/ for Google-Extended
view diff → snapshot