NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
BP bp.com modified +0 −7
  • Removed Disallow: /en_gb/united-kingdom/home/products-and-services/bp-fleet/contactus.html for *
  • Removed Disallow: /en_gb/united-kingdom/home/products-and-services/bp-fleet/legal-notice.html for *
  • Removed Disallow: /en_gb/united-kingdom/home/products-and-services/bp-fleet/privacy-statement.html for *
  • Removed Disallow: /en_gb/united-kingdom/home/references.html for *
  • Removed Disallow: /en_gb/united-kingdom/media/press-releases/bpme-launch.html for *
  • Removed Disallow: /en_gb/united-kingdom/home/references.html for Algolia Crawler
  • … 1 more change(s)
view diff → snapshot
Toyota toyota.com modified +140 −69
  • Removed Disallow: /dealers* for *
  • 🤖 AI Unblocked Amazonbot (Amazon)
  • 🤖 AI Unblocked anthropic-ai (Anthropic)
  • 🤖 AI Unblocked Applebot-Extended (Apple)
  • 🤖 AI Unblocked Bytespider (ByteDance)
  • 🤖 AI Unblocked CCBot (Common Crawl)
  • … 45 more change(s)
view diff → snapshot
Kohl's kohls.com modified +0 −6
  • Removed all rules for user-agent ‘Y!J-ASR’
  • Removed all rules for user-agent ‘YandexBot’
view diff → snapshot
ABM Industries abm.com modified +0 −3
  • Removed Disallow: /es/locations for *
  • Removed Disallow: /locations for *
  • Removed Disallow: /pl/locations for *
view diff → snapshot
Realtor realtor.com modified +5 −1
  • Added Allow: /realestateandhomes-search/*price-na- for *
  • Added Disallow: /realestateandhomes-search/*%2C for *
  • Added Disallow: /realestateandhomes-search/*%2c for *
  • Added Disallow: /realestateandhomes-search/*, for *
  • Added Disallow: /realestateandhomes-search/*/price- for *
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/source-material-manon-macasaet/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-figma-helps-revolut-deliver-wow-for-customers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-manon-macasaet/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-figma-helps-revolut-deliver-wow-for-customers/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-manon-macasaet/$ for Claude-User
  • 🤖 AI Added Allow: /customers/how-figma-helps-revolut-deliver-wow-for-customers/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
AT&T att.com modified +151 −395
  • Added Allow: / for *
  • Added Allow: /.well-known/agent-card.json for *
  • Added Allow: /.well-known/agent-skills/index.json for *
  • Added Allow: /.well-known/mcp/server-card.json for *
  • Added Allow: /ai-instructions/shopping-assistant.md for *
view diff → snapshot
Suno suno.com modified +1 −2
  • Added sitemap https://suno.com/public-site-sitemap/sitemap.xml
  • Removed sitemap https://suno.com/hub-sitemap.xml
  • Removed sitemap https://suno.com/landing-sitemap.xml
view diff → snapshot
The New York Times nytimes.com modified +1 −1
  • Added Allow: /athletic/nfl/game/discuss/ for *
  • Removed Allow: /athletic/mlb/game/discuss/ for *
  • Added Allow: /athletic/nfl/game/discuss/ for Googlebot
  • Removed Allow: /athletic/mlb/game/discuss/ for Googlebot
view diff → snapshot
Figma Community figma.com modified +61 −2
  • 🤖 AI Added Allow: /de-de/loesungen/anzeigen-ersteller/facebook/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-app-icon-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-dateireiniger/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-schaltflaechengenerator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/customers/how-adyen-is-rebuilding-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/creador-de-anuncios/facebook/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot