NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Darden Restaurants darden.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Linde linde.com modified +1 −2
  • Blocked user-agent ‘*’ entirely
  • Removed Disallow: /search for *
  • Removed sitemap https://www.linde.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Darden Restaurants darden.com modified +0 −6
  • Removed Allow: /core/*.avif for *
  • Removed Allow: /core/*.webp for *
  • Removed Allow: /profiles/*.avif for *
  • Removed Allow: /profiles/*.webp for *
  • Removed Disallow: /index.php/search? for *
  • Removed Disallow: /search? for *
view diff → snapshot
Diffbot diffbot.com modified +9 −4
  • Added rules for user-agent ‘*’
  • Added sitemap https://www.diffbot.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/customers/duotone-saves-hours/$ for CHATGPT-User
  • … 22 more change(s)
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://totalenergies.com/sitemap.xml
  • Removed sitemap https://www2.totalenergies.com/sitemap.xml
view diff → snapshot
Tripadvisor tripadvisor.com modified +2 −16
  • Added Disallow: /Games for *
  • Added Disallow: /OwnerResponseApi for *
  • Removed Disallow: /DiscountGroup for *
  • Removed Disallow: /ExitWindow for *
  • Removed Disallow: /ExternalExitWindow for *
  • Removed Disallow: /OwnerResponse for *
  • … 45 more change(s)
view diff → snapshot