NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
The New York Times nytimes.com modified +2 −0
  • Added Allow: /athletic/live-blogs/discuss/ for *
  • Added Allow: /athletic/mlb/game/discuss/ for *
  • Added Allow: /athletic/live-blogs/discuss/ for Googlebot
  • Added Allow: /athletic/mlb/game/discuss/ for Googlebot
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
GE Vernova gevernova.com modified +1 −5
  • Added sitemap https://www.gevernova.com/electrification/sitemap.xml
  • Removed sitemap https://www.gevernova.com/grid-solutions/sitemap.xml
  • Removed sitemap https://www.gevernova.com/hydropower/sitemap.xml
  • Removed sitemap https://www.gevernova.com/power-conversion/sitemap.xml
  • Removed sitemap https://www.gevernova.com/solar-storage/sitemap.xml
  • Removed sitemap https://www.lmwindpower.com/sitemap.xml
view diff → snapshot
CHS chsinc.com modified +7 −6
  • Removed all rules for user-agent ‘facebookexternalhit’
  • Added Disallow: (empty) for *
  • Removed Disallow: /*?* for *
  • Removed Disallow: /-/media/ for *
  • Crawl-delay for *: none → 10
  • Added sitemap https://www.chsinc.com/sitemap_index.xml
  • … 1 more change(s)
view diff → snapshot
Claude claude.ai modified +1 −8
  • Added Allow: /api/desktop/ for *
  • Removed Disallow: /chat/* for *
  • Removed Disallow: /join/* for *
  • Removed Disallow: /magic-link* for *
  • Removed Disallow: /new?* for *
  • Removed Disallow: /onboarding* for *
  • … 3 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −12
  • Added Allow: /branded-story/ for ChatGPT Operator
  • Removed Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Removed Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /branded-story/ for ChatGPT-User
  • 🤖 AI Removed Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Removed Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Corteva corteva.com modified +1 −2
  • Added Disallow: /content/dam/dpagco/ for *
  • Removed Disallow: /content/dam/dpagco/corteva/global/corporate/files/code-of-conduct/Corteva_Code_Interactive_ruRU_Russian.pdf for *
  • Removed Disallow: /content/dam/dpagco/corteva/global/corporate/general/files/code-of-conduct/Corteva%20QRG%20Russian.pdf for *
view diff → snapshot
Shell shell.com modified +2 −0
  • Added Disallow: /terms-of-use/shell-motorsports-ferrari-giveaway-activity-terms-and-conditions.html for *
  • Added Disallow: /terms-of-use/shell-motorsports-ferrari-giveaway-activity-terms-and-conditions/ for *
view diff → snapshot
Jabil jabil.com modified +5 −1
  • Added Allow: /.resources/jabil-main/themes/jabil-one-26/src/img/favicon/ for *
view diff → snapshot
Canva canva.com modified +2 −0
  • 🤖 AI Added Allow: /*/design-school/ for anthropic-ai
  • 🤖 AI Added Allow: /design-school/ for anthropic-ai
  • 🤖 AI Added Allow: /*/design-school/ for Applebot-Extended
  • 🤖 AI Added Allow: /design-school/ for Applebot-Extended
  • 🤖 AI Added Allow: /*/design-school/ for CCBot
  • 🤖 AI Added Allow: /design-school/ for CCBot
  • … 6 more change(s)
view diff → snapshot