NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
All changes 🤖 AI-crawler changes Hiding initial archives
Darden Restaurants darden.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Crown Holdings crowncork.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Realtor realtor.com modified +4 −0
  • Added Disallow: /advice/search/ for *
  • Added Disallow: /guides/search/ for *
  • Added Disallow: /living/search/ for *
  • Added Disallow: /news/search/ for *
view diff → snapshot
Slate slate.com modified +3 −0
  • Added rules for user-agent ‘UpContent’
  • Added rules for user-agent ‘UpContent-QA’
  • Added rules for user-agent ‘UpContent-Test’
view diff → snapshot
Fortra fortra.com modified +1 −1
  • 🤖 AI Unblocked Google-Extended (Google)
  • 🤖 AI Added Allow: / for Google-Extended
view diff → snapshot
Weather weather.com modified +1 −4
  • Removed Disallow: /*?base for CriteoBot/0.1
  • Removed Disallow: /*?phenomena for CriteoBot/0.1
  • Removed Disallow: /*?traffic_source for CriteoBot/0.1
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +40 −0
  • Blocked user-agent ‘ahrefsbot’ entirely
  • Blocked user-agent ‘cazoodlebot’ entirely
  • Blocked user-agent ‘dotbot/1.0’ entirely
  • Blocked user-agent ‘gigabot’ entirely
  • Blocked user-agent ‘mj12bot’ entirely
  • Blocked user-agent ‘sogou inst spider’ entirely
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
HP hp.com modified +8 −1
  • Added sitemap https://www.hp.com/in-en/shop/media/facets_url/facets_url_in_en.txt
  • Added sitemap https://www.hp.com/in-en/shop/media/facets_url/ink_url_in_en.txt
  • Added sitemap https://www.hp.com/in-en/shop/media/facets_url/products_url_in_en.txt
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +105 −1
  • 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-design-tools/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot