NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2647snapshots
1476changes detected
231this week
All changes 🤖 AI-crawler changes Showing initial archives
Le Figaro lefigaro.fr modified +58 −0
  • Blocked user-agent ‘CohereBot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘GeistHaus-PageFetcher’ entirely
  • Blocked user-agent ‘Kimi-SearchBot’ entirely
  • Blocked user-agent ‘Kimi-User’ entirely
  • Blocked user-agent ‘KimiBot’ entirely
  • … 3 more change(s)
view diff → snapshot
Microsoft microsoft.com modified +3 −1
  • Added Disallow: /store/XboxComMsCom3PAdsOptOutCookieSync.html for *
  • Added Disallow: /store/XboxComMsComCartMuidSync.html for *
view diff → snapshot
Rivian Automotive rivian.com modified +7 −3
  • Added Disallow: /account/api/ for *
  • Added Disallow: /auth/api/ for *
  • Added Disallow: /demo-drive/api/ for *
  • Added Disallow: /experience/api/ for *
  • Added Disallow: /quad/api/ for *
  • Added Disallow: /root/api/ for *
  • … 4 more change(s)
view diff → snapshot
Labcorp labcorp.com modified +12 −58
  • Added Disallow: /index.php/admin/ for *
  • Added Disallow: /index.php/comment/reply/ for *
  • Added Disallow: /index.php/filter/tips for *
  • Added Disallow: /index.php/node/add/ for *
  • Added Disallow: /index.php/user/ for *
  • Removed Allow: /content/dam/ for *
  • … 37 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Kraken kraken.com modified +0 −1
  • Removed sitemap https://www.kraken.com/sitemaps/embed-convert-buy.xml
view diff → snapshot
Figma Community figma.com modified +15 −0
  • 🤖 AI Added Allow: /de-de/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/principles-in-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Wired wired.com modified +3 −1
  • Added Allow: /*?*utm_medium=social for *
  • Added Allow: /*?ref= for *
view diff → snapshot
Pinterest pinterest.com modified +39 −41
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideahub_christmas_sitemap_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideahub_new_years_eve_sitemap_www.pinterest.com.xml
view diff → snapshot
Rockwell Automation rockwellautomation.com modified +0 −53
  • Removed Disallow: /de-at/docs for *
  • Removed Disallow: /de-ch/docs for *
  • Removed Disallow: /de-de/docs for *
  • Removed Disallow: /en-au/docs for *
  • Removed Disallow: /en-be/docs for *
  • Removed Disallow: /en-ca/docs for *
  • … 39 more change(s)
view diff → snapshot