NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
Hermes hermes.com modified +39 −42
  • Removed Allow: /*/*/user-guides/$ for *
  • Removed Disallow: /*/*/user-guides/* for *
  • Added sitemap https://www.hermes.com/at/de/sitemap_index.xml
  • Added sitemap https://www.hermes.com/au/en/sitemap_index.xml
  • Added sitemap https://www.hermes.com/be/en/sitemap_index.xml
  • Added sitemap https://www.hermes.com/be/fr/sitemap_index.xml
  • … 45 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +15 −0
  • Added rules for user-agent ‘Amzn-SearchBot’
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • Added Allow: / for *
  • … 1 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −0
  • 🤖 AI Added Allow: /de-de/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • … 24 more change(s)
view diff → snapshot
Indian Institute of Technology Bombay iitb.ac.in created +13 −1
  • Added Disallow: /*/media/oembed for *
  • Added Disallow: /README.md for *
  • Added Disallow: /composer/Metapackage/README.txt for *
  • Added Disallow: /composer/Plugin/ProjectMessage/README.md for *
  • Added Disallow: /composer/Plugin/Scaffold/README.md for *
  • Added Disallow: /composer/Plugin/VendorHardening/README.txt for *
  • … 8 more change(s)
view diff → snapshot
Cohere cohere.com modified +2 −0
  • Added Disallow: /mcp-expertise for *
  • Added Disallow: /mcp-landscape for *
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
France (Gouvernement) gouvernement.fr modified +0 −2
  • Removed sitemap https://www.info.gouv.fr/engagements-sitemap.xml
  • Removed sitemap https://www.info.gouv.fr/local_priority_policies-sitemap.xml
view diff → snapshot
Hacker News news.ycombinator.com modified +4 −0
  • Added Disallow: /favex? for *
  • Added Disallow: /loginx for *
  • Added Disallow: /replyx? for *
  • Added Disallow: /votex? for *
view diff → snapshot
NBCUniversal nbcuniversal.com modified +0 −9
  • Removed all rules for user-agent ‘* Disallow:’
  • Removed Allow: /newsroom for *
  • Removed Allow: /newsroom/* for *
  • Removed Disallow: /README.txt for *
  • Removed sitemap https://www.nbcuniversal.com/sitemap.xml
view diff → snapshot