NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Sempra sempra.com modified +78 −58
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +60 −56
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261204_international_day_of_banks_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_chanukah_hanukkah_first_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_first_day_of_hanukkah_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_international_volunteer_day_for_economic_and_socia_www.pinterest.com.xml
view diff → snapshot
Airbnb airbnb.com modified +24 −134
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘MistralAI-Training’ entirely
  • Blocked user-agent ‘Webzio-Extended’ entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Removed Allow: /.well-known/amphtml/apikey.pub for ClaudeBot
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
C.H. Robinson chrobinson.com modified +6 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • Added Disallow: Amazonbot for *
  • Added Disallow: Amzn-SearchBot for *
  • Added Disallow: Bytespider for *
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-aitana-pico/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Shell shell.com modified +14 −4
  • Added Disallow: /shellenergy/othersolutions/carbonmarketreports/shell-voluntary-carbon-market-reports.html for *
  • Added Disallow: /shellenergy/othersolutions/carbonmarketreports/shell-voluntary-carbon-market-reports/ for *
  • Added Disallow: /shellenergy/othersolutions/environmental-products/contact-us.html for *
  • Added Disallow: /shellenergy/othersolutions/environmental-products/contact-us/ for *
  • Added Disallow: /shellenergy/othersolutions/environmental-products/duedilligence.html for *
  • Added Disallow: /shellenergy/othersolutions/environmental-products/duedilligence/ for *
  • … 12 more change(s)
view diff → snapshot