NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
ResearchGate researchgate.net modified +5 −0
  • Blocked user-agent ‘GoogleOther’ entirely
  • Blocked user-agent ‘GoogleOther-Image’ entirely
  • Blocked user-agent ‘GoogleOther-Video’ entirely
view diff → snapshot
Glencore glencore.com modified +0 −4
  • Removed Disallow: /.rest/api/v1/documents/1ab16a3be154f86e1478a8f0e6277f65/D%C3%A9ploiement+de+Coupa%21+_+3+novembre+2025.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/2c4c65ce2237cc9d2efdd214ec1b85df/We+are+Live%21+_+November+3%2C+2025.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/535eca4046bad6d3dec0a5c608299fd0/Inscription+et+Formation+Portail+Fournisseur+Coupa-+8+septembre+2025.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/e33dab157f5c29da07a1873996829885/Informations+Importantes+-+Prochain+Lancement+de+Coupa+-+2+Juillet+2025+%281%29.pdf for *
view diff → snapshot
Zhongji Innolight innolight.com modified +3 −13
  • Added Allow: / for *
  • Removed Disallow: /bin/ for *
  • Removed Disallow: /css/ for *
  • Removed Disallow: /demo/ for *
  • Removed Disallow: /extend/ for *
  • Removed Disallow: /fonts/ for *
  • … 9 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +3 −70
  • Added Allow: / for *
  • Added Disallow: /promotions/emails/ for *
  • Added Disallow: /sponsored/travel/msc-cruises/ for *
  • 🤖 AI Removed Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
  • 🤖 AI Removed Allow: /bespoke-content/ for AI2Bot
  • 🤖 AI Removed Allow: /betting/ for AI2Bot
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Bilibili bilibili.com modified +19 −0
  • Added rules for user-agent ‘Discordbot’
  • Added rules for user-agent ‘facebookexternalhit’
  • Added rules for user-agent ‘Facebot’
  • Added rules for user-agent ‘LinkedInBot’
  • Added rules for user-agent ‘Slackbot’
  • Added rules for user-agent ‘TelegramBot’
  • … 10 more change(s)
view diff → snapshot
Hyundai Motor hyundai.com modified +37 −0
  • Added rules for user-agent ‘CazoodleBot’
  • Added rules for user-agent ‘dotbot’
  • Added rules for user-agent ‘dotbot/1.0’
  • Added rules for user-agent ‘Gigabot’
  • Added rules for user-agent ‘ia_archiver’
  • Added rules for user-agent ‘MJ12bot’
  • … 24 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Slate slate.com modified +9 −0
  • Added rules for user-agent ‘AdpResearchBot’
  • Blocked user-agent ‘artsdata-crawler’ entirely
  • Added rules for user-agent ‘BuiltWith’
  • Blocked user-agent ‘ElevenlabsCrawler’ entirely
  • Added rules for user-agent ‘FlyingPress’
  • Blocked user-agent ‘Quartr Crawler’ entirely
  • … 3 more change(s)
view diff → snapshot
Bloomberg bloomberg.com modified +43 −255
  • 🤖 AI Added Allow: /?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa/?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa/llms.txt/?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa/llms.txt?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /africa?bloomberg_debug= for ChatGPT-User
  • 🤖 AI Added Allow: /asia/?bloomberg_debug= for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot
Adidas adidas.com modified +1 −3
  • Removed sitemap https://www.adidas.com/bh/ar/sitemap
  • Removed sitemap https://www.adidas.com/bh/en/sitemap
view diff → snapshot
LKQ lkqcorp.com modified +2 −0
  • Added Disallow: /*&s= for *
  • Added Disallow: /*?s= for *
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
Danaher danaher.com modified +6 −1
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
  • … 1 more change(s)
view diff → snapshot