NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2571snapshots
1398changes detected
225this week
All changes 🤖 AI-crawler changes Showing initial archives
Reliance Steel & Aluminum rsac.com modified +14 −1
  • Added Disallow: (empty) for *
  • Added Disallow: /wp-content/uploads/wpforms/ for *
  • Removed Disallow: /policies/ for *
  • Added sitemap https://www.reliance.com/sitemap_index.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/loesungen/farbe-aendern/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-design-assistent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/dm-sans/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/solutions/ai-change-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/solutions/ai-design-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/solutions/ai-design-assistant/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
The Trade Desk thetradedesk.com modified +1 −1
  • Added sitemap https://www.thetradedesk.cn/sitemap.xml
  • Removed sitemap https://www.thetradedesk.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Nubank nubank.com.br modified +42 −54
  • Removed Disallow: /nu/conta/caixinha-turbo for *
  • Removed Disallow: /nu/investimentos/caixinhas for *
  • Removed Disallow: /nu/conta/caixinha-turbo for DatoCmsSearchBot
  • Removed Disallow: /nu/investimentos/caixinhas for DatoCmsSearchBot
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-bildbearbeitung/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/changer-couleur-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/retouche-image-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/vectoriser-image/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +35 −33
  • Added sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_priority_segment_2026q3_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/priority_segment_2026q3_www.pinterest.com.xml
view diff → snapshot
Zscaler zscaler.com modified +24 −1
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 45 more change(s)
view diff → snapshot
Tractor Supply tractorsupply.com modified +2 −9
  • Added Disallow: /gtwy/accountservice/orchestration/accounts/* for *
  • Added Disallow: /gtwy/accountservice/orchestration/signin/* for *
  • Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/change/password for *
  • Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/forgot/password for *
  • Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/lookup for *
  • Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/update for *
  • … 5 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot