NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/the-power-of-product-based-learning/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +36 −25
  • Added sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_seasonal_halloween_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261121_world_television_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261122_albanian_alphabet_day_albanian_community_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261122_alphabet_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261122_christ_the_king_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261123_labor_thanksgiving_day_www.pinterest.com.xml
  • … 5 more change(s)
view diff → snapshot
Reuters reuters.com modified +1 −0
  • Added Disallow: /*/site-search/ for AASA-Bot
  • Added Disallow: /*/site-search/ for ADmantX
  • Added Disallow: /*/site-search/ for AdmiralBot
  • Added Disallow: /*/site-search/ for AdsBot-Google
  • Added Disallow: /*/site-search/ for AdsBot-Google-Mobile
  • Added Disallow: /*/site-search/ for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot
Nordea nordea.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Daily Mail dailymail.co.uk modified +1 −2
  • Removed Allow: /buyline/ for AhrefsBot
  • 🤖 AI Removed Allow: /buyline/ for AI2Bot
  • Removed Allow: /buyline/ for Ai2Bot-Dolma
  • Removed Allow: /buyline/ for amazon-kendra-
  • 🤖 AI Removed Allow: /buyline/ for Amazonbot
  • 🤖 AI Removed Allow: /buyline/ for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +10 −0
  • 🤖 AI Added Allow: /es-es/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/double-click-does-efficiency-kill-love/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/reports/jp-developers-field-guide-ai-era/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/reports/kr-developers-field-guide-ai-era/$ for CHATGPT-User
  • … 44 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot