NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2643snapshots
1471changes detected
226this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/build-with-joy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/build-with-joy/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/build-with-joy/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Volvo Cars volvocars.com modified +7 −1
  • Added rules for user-agent ‘Googlebot’
  • Added Disallow: /*/build/ for YisouSpider
  • Removed Disallow: /pl/build/ for YisouSpider
view diff → snapshot
HP hp.com modified +79 −9
  • Added Disallow: /ar-es/shop/banner/ajax/load/ for *
  • Added Disallow: /ar-es/shop/hpols_catalog/ for *
  • Added Disallow: /au-en/shop/banner/ajax/load/ for *
  • Added Disallow: /au-en/shop/hpols_catalog/ for *
  • Added Disallow: /br-pt/shop/banner/ajax/load/ for *
  • Added Disallow: /br-pt/shop/hpols_catalog/ for *
  • … 32 more change(s)
view diff → snapshot
Figma Community figma.com modified +68 −0
  • 🤖 AI Added Allow: /de-de/blog/introducing-keyboard-accessibility-features/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/new-features-in-figma-buzz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/5-shifts-redefining-design-systems-in-the-ai-era/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/you-never-stop-cultivating-taste/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Ukraine (KMU) kmu.gov.ua modified +1 −12
  • Added Disallow: /storage/ for *
  • Removed Disallow: *utm= for *
  • Removed Disallow: /api/ for *
  • Removed Disallow: /backend/ for *
  • Removed Disallow: /timeline for *
  • Crawl-delay for *: 1 → none
  • … 1 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +7 −6
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_www.pinterest.com.xml
view diff → snapshot
Reuters reuters.com modified +1 −0
  • Added sitemap https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml
view diff → snapshot
ICICI Bank icicibank.com modified +2 −0
  • Added Disallow: /j.php for *
  • Added Disallow: /personal-banking/account-online/power-pay-account/ for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Kuehne+Nagel kuehne-nagel.com modified +33 −1
  • Added rules for user-agent ‘amazon-Qbusiness’
  • Added rules for user-agent ‘Applebot’
  • Added rules for user-agent ‘Baiduspider’
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • Added rules for user-agent ‘SERankingBacklinksBot’
  • … 4 more change(s)
view diff → snapshot