NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2586snapshots
1413changes detected
240this week
All changes 🤖 AI-crawler changes Showing initial archives
Goodyear Tire & Rubber goodyear.com modified +40 −0
  • Blocked user-agent ‘ahrefsbot’ entirely
  • Blocked user-agent ‘cazoodlebot’ entirely
  • Blocked user-agent ‘dotbot/1.0’ entirely
  • Blocked user-agent ‘gigabot’ entirely
  • Blocked user-agent ‘mj12bot’ entirely
  • Blocked user-agent ‘sogou inst spider’ entirely
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
HP hp.com modified +8 −1
  • Added sitemap https://www.hp.com/in-en/shop/media/facets_url/facets_url_in_en.txt
  • Added sitemap https://www.hp.com/in-en/shop/media/facets_url/ink_url_in_en.txt
  • Added sitemap https://www.hp.com/in-en/shop/media/facets_url/products_url_in_en.txt
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +105 −1
  • 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-design-tools/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Darden Restaurants darden.com modified +0 −6
  • Removed Allow: /core/*.avif for *
  • Removed Allow: /core/*.webp for *
  • Removed Allow: /profiles/*.avif for *
  • Removed Allow: /profiles/*.webp for *
  • Removed Disallow: /index.php/search? for *
  • Removed Disallow: /search? for *
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
South China Morning Post scmp.com modified +122 −154
  • Removed all rules for user-agent ‘Googlebot’
  • Added Disallow: /*?*campaign=* for *
  • Added Disallow: /*?*cid=* for *
  • Added Disallow: /*?*nochartbeat=* for *
  • Added Disallow: /*?*noga=* for *
  • Added Disallow: /*?*nograpeshot=* for *
  • … 37 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Oneok oneok.com removed +0 −3
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +0 −93
  • 🤖 AI Removed Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /es-es/blog/7-questions-from-lc/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
AliExpress aliexpress.com modified +0 −7
  • Removed all rules for user-agent ‘AdsBot-Google’
  • Removed all rules for user-agent ‘AdsBot-Google-Mobile’
  • Removed Disallow: /wholesale* for *
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/e-books/pfpt_jp_sb_ciso_buyers_guide.pdf for *
view diff → snapshot