NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
GoDaddy godaddy.com modified +23 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • 🤖 AI Added Allow: /domain-auctions/ for Claude-User
view diff → snapshot
Hyundai Motor hyundai.com modified +19 −8
  • Added sitemap https://www.hyundai.com/cz/cz/cci-sitemap.xml
  • Added sitemap https://www.hyundai.com/cz/cz/sitemap.xml
  • Added sitemap https://www.hyundai.com/de/de/cci-sitemap.xml
  • Added sitemap https://www.hyundai.com/es/es/cci-sitemap.xml
  • Added sitemap https://www.hyundai.com/fr/fr/cci-sitemap.xml
  • Added sitemap https://www.hyundai.com/it/it/cci-sitemap.xml
  • … 5 more change(s)
view diff → snapshot
Weather weather.com modified +20 −133
  • Added rules for user-agent ‘Brightbot 1.0’
  • Blocked user-agent ‘Google-CloudVertexBot’ entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Claude-Web (Anthropic) - rules removed
  • … 45 more change(s)
view diff → snapshot
Sony Music sonymusic.com modified +7 −60
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 6 more change(s)
view diff → snapshot
Expeditors International expeditors.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Tenet Healthcare tenethealth.com modified +61 −53
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
  • Added rules for user-agent ‘bingbot’
  • Added rules for user-agent ‘DuckDuckBot’
  • Added rules for user-agent ‘facebookexternalhit’
  • … 22 more change(s)
view diff → snapshot
Guyana (Office of the President) op.gov.gy modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Costa Rica (Presidencia) presidencia.go.cr modified +70 −59
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for PerplexityBot
view diff → snapshot
EPFL epfl.ch modified +0 −61
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • … 4 more change(s)
view diff → snapshot
AI21 Labs ai21.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Iceland (Government.is) government.is modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Ralph Lauren ralphlauren.com modified +5 −0
  • Added Disallow: *brands*/r/* for *
  • Added Disallow: *cg*/r/* for *
  • Added Disallow: *customize*/r/* for *
  • Added Disallow: *hidden*/r/* for *
  • Added Disallow: *new-arrivals*/r/* for *
view diff → snapshot