NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/customers/duotone-saves-hours/$ for CHATGPT-User
  • … 22 more change(s)
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://totalenergies.com/sitemap.xml
  • Removed sitemap https://www2.totalenergies.com/sitemap.xml
view diff → snapshot
Tripadvisor tripadvisor.com modified +2 −16
  • Added Disallow: /Games for *
  • Added Disallow: /OwnerResponseApi for *
  • Removed Disallow: /DiscountGroup for *
  • Removed Disallow: /ExitWindow for *
  • Removed Disallow: /ExternalExitWindow for *
  • Removed Disallow: /OwnerResponse for *
  • … 45 more change(s)
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for PerplexityBot
view diff → snapshot
BBC bbc.com modified +7 −1
  • 🤖 AI Added Allow: /storyworks for ChatGPT-User
  • 🤖 AI Added Allow: /storyworks/ for ChatGPT-User
  • 🤖 AI Added Allow: /storyworks for Google-Extended
  • 🤖 AI Added Allow: /storyworks/ for Google-Extended
  • 🤖 AI Added Allow: /storyworks for GPTBot
  • 🤖 AI Added Allow: /storyworks/ for GPTBot
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://www2.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
Continental continental.com modified +32 −1
  • Blocked user-agent ‘AmazonProductDiscovery’ entirely
  • Added Disallow: /*&tx_condownloadcenter_api* for *
  • Added Disallow: /*&tx_conevents_list* for *
  • Added Disallow: /*&tx_conrssfeed_list* for *
  • Added Disallow: /*&tx_consecuredarea_* for *
  • Added Disallow: /*&tx_solr* for *
  • … 5 more change(s)
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Lithia Motors lithiamotors.com modified +61 −1
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • … 5 more change(s)
view diff → snapshot