NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Corriere della Sera corriere.it modified +6 −0
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘NovriumRanker’ entirely
  • Blocked user-agent ‘Stucco5000’ entirely
view diff → snapshot
Yandex yandex.com modified +15 −2
  • Added Disallow: /games/*?*clid= for *
  • Added Disallow: /games/*?*deviceid= for *
  • Added Disallow: /games/*?*flags= for *
  • Added Disallow: /games/*?*inapp_id= for *
  • Added Disallow: /games/*?*k50id= for *
  • Added Disallow: /games/*?*lang= for *
  • … 11 more change(s)
view diff → snapshot
Vimeo vimeo.com modified +45 −21
  • 🤖 AI Added Allow: /de/product/ for anthropic-ai
  • 🤖 AI Added Allow: /de/video-uploader for anthropic-ai
  • 🤖 AI Added Allow: /es/product/ for anthropic-ai
  • 🤖 AI Added Allow: /es/video-uploader for anthropic-ai
  • 🤖 AI Added Allow: /fr/product/ for anthropic-ai
  • 🤖 AI Added Allow: /fr/video-uploader for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
ElevenLabs elevenlabs.io modified +2 −0
  • Added Allow: /docs/api/fern-docs/og for *
  • Added Allow: /docs/api/fern-docs/og for Twitterbot
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Arrow Electronics arrow.com removed +0 −102
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /fr-fr/fonts/geometric/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/humanist/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/neo-grotesque/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/rounded/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/script/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/geometric/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Ansys ansys.com modified +10 −14
  • Removed Disallow: /*.pdf$ for *
  • Removed Disallow: /home-page-redesign for *
  • Removed Disallow: /new-redesign for *
  • Removed Disallow: /unsupported-browser-notice/* for *
  • Added sitemap https://ansys.synopsys.com/de-de/sitemap.xml
  • Added sitemap https://ansys.synopsys.com/en-gb/sitemap.xml
  • … 18 more change(s)
view diff → snapshot
Shutterstock shutterstock.com modified +16 −8
  • Added Disallow: /*/search/personalized/ for *
  • Added Disallow: /*/search/similar/ for *
  • Added Disallow: /search/personalized/ for *
  • Added Disallow: /search/similar/ for *
  • Removed Disallow: /search/personalized/* for *
  • Removed Disallow: /search/similar/* for *
  • … 18 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Uol uol.com.br modified +4 −0
  • Added Disallow: /*.ghtm?*&uol_app for *
  • Added Disallow: /*.ghtm?uol_app for *
  • Added Disallow: /*.htm?*&uol_app for *
  • Added Disallow: /*.htm?uol_app for *
view diff → snapshot