NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2586snapshots
1413changes detected
240this week
All changes 🤖 AI-crawler changes Showing initial archives
BBC bbc.com modified +7 −1
  • 🤖 AI Added Allow: /storyworks for ChatGPT-User
  • 🤖 AI Added Allow: /storyworks/ for ChatGPT-User
  • 🤖 AI Added Allow: /storyworks for Google-Extended
  • 🤖 AI Added Allow: /storyworks/ for Google-Extended
  • 🤖 AI Added Allow: /storyworks for GPTBot
  • 🤖 AI Added Allow: /storyworks/ for GPTBot
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://www2.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
Continental continental.com modified +32 −1
  • Blocked user-agent ‘AmazonProductDiscovery’ entirely
  • Added Disallow: /*&tx_condownloadcenter_api* for *
  • Added Disallow: /*&tx_conevents_list* for *
  • Added Disallow: /*&tx_conrssfeed_list* for *
  • Added Disallow: /*&tx_consecuredarea_* for *
  • Added Disallow: /*&tx_solr* for *
  • … 5 more change(s)
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Lithia Motors lithiamotors.com modified +61 −1
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • … 5 more change(s)
view diff → snapshot
Darden Restaurants darden.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Crown Holdings crowncork.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Realtor realtor.com modified +4 −0
  • Added Disallow: /advice/search/ for *
  • Added Disallow: /guides/search/ for *
  • Added Disallow: /living/search/ for *
  • Added Disallow: /news/search/ for *
view diff → snapshot
Slate slate.com modified +3 −0
  • Added rules for user-agent ‘UpContent’
  • Added rules for user-agent ‘UpContent-QA’
  • Added rules for user-agent ‘UpContent-Test’
view diff → snapshot
Fortra fortra.com modified +1 −1
  • 🤖 AI Unblocked Google-Extended (Google)
  • 🤖 AI Added Allow: / for Google-Extended
view diff → snapshot
Weather weather.com modified +1 −4
  • Removed Disallow: /*?base for CriteoBot/0.1
  • Removed Disallow: /*?phenomena for CriteoBot/0.1
  • Removed Disallow: /*?traffic_source for CriteoBot/0.1
view diff → snapshot