NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Orange orange.com modified +2 −2
  • Removed Disallow: /en/contact-email/* for *
  • Removed Disallow: /fr/contact-email/* for *
view diff → snapshot
CNN cnn.com modified +5 −0
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Firecrawl’ entirely
  • Blocked user-agent ‘FirecrawlAgent’ entirely
  • Blocked user-agent ‘NimbleCrawler’ entirely
  • Blocked user-agent ‘TavilyBot’ entirely
view diff → snapshot
Sophos sophos.com modified +7 −1
  • Added rules for user-agent ‘Amzn-SearchBot’
  • Added rules for user-agent ‘Amzn-User’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Removed Allow: / for Amazonbot
view diff → snapshot
Exa exa.ai modified +62 −0
  • Added Allow: /library$ for *
  • Added Allow: /library/$ for *
  • Added Allow: /library/earnings-call$ for *
  • Added Allow: /library/earnings-call/$ for *
  • Added Allow: /library/earnings-call? for *
  • Added Allow: /library/event$ for *
  • … 45 more change(s)
view diff → snapshot
Boston Scientific bostonscientific.com modified +2 −1
  • Added sitemap https://www.bostonscientific.com/us/en/healthcare-professionals/products/pdp/sitemap.xml
  • Added sitemap https://www.bostonscientific.com/us/en/healthcare-professionals/products/plp/sitemap.xml
  • Removed sitemap https://www.bostonscientific.com/us/en/healthcare-professionals/products/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for PerplexityBot
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-us-eb-new-rules-of-business-communication-governance.pdf for *
view diff → snapshot
Thyssenkrupp thyssenkrupp.com modified +6 −1
  • Added Disallow: *UCPthyssenkruppAG* for *
  • Added Disallow: /_api for *
  • Added Disallow: /_media/ for *
  • Added Disallow: /assets.files/ for *
  • Added sitemap https://www.thyssenkrupp.com/sitemap.xml
view diff → snapshot
Daily Mail dailymail.co.uk modified +2 −1
  • Added Allow: /llms.txt for AhrefsBot
  • 🤖 AI Added Allow: /llms.txt for AI2Bot
  • Added Allow: /llms.txt for Ai2Bot-Dolma
  • Added Allow: /llms.txt for amazon-kendra-
  • 🤖 AI Added Allow: /llms.txt for Amazonbot
  • 🤖 AI Added Allow: /llms.txt for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
SAP sap.com modified +3 −1
  • Added sitemap https://www.sap.com/sitemap-documents-index.xml
  • Added sitemap https://www.sap.com/sitemap-videos-index.xml
view diff → snapshot