NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2634snapshots
1462changes detected
251this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Hulu hulu.com modified +132 −207
  • Added rules for user-agent ‘360Spider’
  • Added rules for user-agent ‘Adidxbot’
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • … 45 more change(s)
view diff → snapshot
Le Figaro lefigaro.fr modified +58 −0
  • Blocked user-agent ‘CohereBot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘GeistHaus-PageFetcher’ entirely
  • Blocked user-agent ‘Kimi-SearchBot’ entirely
  • Blocked user-agent ‘Kimi-User’ entirely
  • Blocked user-agent ‘KimiBot’ entirely
  • … 3 more change(s)
view diff → snapshot
Microsoft microsoft.com modified +3 −1
  • Added Disallow: /store/XboxComMsCom3PAdsOptOutCookieSync.html for *
  • Added Disallow: /store/XboxComMsComCartMuidSync.html for *
view diff → snapshot
Rivian Automotive rivian.com modified +7 −3
  • Added Disallow: /account/api/ for *
  • Added Disallow: /auth/api/ for *
  • Added Disallow: /demo-drive/api/ for *
  • Added Disallow: /experience/api/ for *
  • Added Disallow: /quad/api/ for *
  • Added Disallow: /root/api/ for *
  • … 4 more change(s)
view diff → snapshot
Labcorp labcorp.com modified +12 −58
  • Added Disallow: /index.php/admin/ for *
  • Added Disallow: /index.php/comment/reply/ for *
  • Added Disallow: /index.php/filter/tips for *
  • Added Disallow: /index.php/node/add/ for *
  • Added Disallow: /index.php/user/ for *
  • Removed Allow: /content/dam/ for *
  • … 37 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Kraken kraken.com modified +0 −1
  • Removed sitemap https://www.kraken.com/sitemaps/embed-convert-buy.xml
view diff → snapshot
Figma Community figma.com modified +15 −0
  • 🤖 AI Added Allow: /de-de/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/principles-in-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Wired wired.com modified +3 −1
  • Added Allow: /*?*utm_medium=social for *
  • Added Allow: /*?ref= for *
view diff → snapshot
Pinterest pinterest.com modified +39 −41
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideahub_christmas_sitemap_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideahub_new_years_eve_sitemap_www.pinterest.com.xml
view diff → snapshot