NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /webinars/building-your-first-component-with-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/connecting-reviewing-and-shipping-your-component/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/designer-to-builder-series/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/getting-ready-to-build-from-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/building-your-first-component-with-ai/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/connecting-reviewing-and-shipping-your-component/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-us-eb-the-new-rules-of-business-communication-governance.pdf for *
view diff → snapshot
Copart copart.com modified +39 −1
  • Added rules for user-agent ‘*’
  • Added sitemap https://www.copart-sitemaps.com/ar/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/es/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/fr-CA/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/pl/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/ru/sitemap-index.xml
  • … 1 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Nubank nubank.com.br modified +0 −6
  • Removed Disallow: /nubank-parque for *
  • Removed Disallow: /nubank-parque for DatoCmsSearchBot
view diff → snapshot
Figma Community figma.com modified +13 −0
  • 🤖 AI Added Allow: /blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/start-anywhere/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Snyk snyk.io modified +0 −2
  • Removed sitemap https://snyk.io/advisor/sitemap/sitemap-index.xml
  • Removed sitemap https://snyk.io/sitemaps/sitemap-misc.xml
view diff → snapshot
Threads threads.net modified +1 −0
  • Added sitemap https://www.threads.com/sitemap/threads_high_potential_posts_sitemap.xml.gz
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot