NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2601snapshots
1428changes detected
255this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for PerplexityBot
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/loesungen/ki-assistent-design-uebergabe/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-preistabellen-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/asistente-de-entrega-de-disenos-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-figma-a-codigo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-mockup-a-prototipo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-sketch-a-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Runway runwayml.com modified +1 −1
  • Added sitemap https://runway.com/sitemap.xml
  • Removed sitemap https://runwayml.com/sitemap.xml
view diff → snapshot
Reuters reuters.com modified +2 −1
  • Added Disallow: /account/subscribe/payment for AASA-Bot
  • Added Disallow: /jp/ for AASA-Bot
  • Removed Disallow: /account/subscribe/payment/ for AASA-Bot
  • Added Disallow: /account/subscribe/payment for ADmantX
  • Added Disallow: /jp/ for ADmantX
  • Removed Disallow: /account/subscribe/payment/ for ADmantX
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Evonik evonik.com modified +2 −0
  • Added Disallow: /*j_security_check for *
  • Added Disallow: /content/dam/ for *
view diff → snapshot
DHL Group dhl.com modified +0 −1
  • Removed Disallow: /lu-de/home/global-forwarding/help-center/claims-complaints-compliments-form.html for *
view diff → snapshot
AbbVie abbvie.com modified +3 −0
  • Added Disallow: /content/dam/abbvie-com2/pdfs/joinus for *
  • Added Disallow: /content/dam/abbvie-com2/pdfs/joinus/2025-total-rewards-brochure-aug2025.pdf for *
  • Added Disallow: /content/dam/abbvie-com2/pdfs/joinus/2026-abv-total-rewards-resources-guide-jan-publish.pdf for *
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +40 −0
  • Blocked user-agent ‘ahrefsbot’ entirely
  • Blocked user-agent ‘cazoodlebot’ entirely
  • Blocked user-agent ‘dotbot/1.0’ entirely
  • Blocked user-agent ‘gigabot’ entirely
  • Blocked user-agent ‘mj12bot’ entirely
  • Blocked user-agent ‘sogou inst spider’ entirely
  • … 20 more change(s)
view diff → snapshot