NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2625snapshots
1453changes detected
242this week
All changes 🤖 AI-crawler changes Hiding initial archives
MongoDB mongodb.com modified +2 −1
  • Added Disallow: /company/blog/search for *
  • Added Disallow: /docs/site-search/ for *
  • Removed Disallow: /docs/search/ for *
view diff → snapshot
Huntington Ingalls Industries hii.com modified +70 −3
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 9 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Slate slate.com modified +20 −0
  • Added rules for user-agent ‘AdmixerBot’
  • Added rules for user-agent ‘AffsignalCrawler’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘CohereBot’ entirely
  • Added rules for user-agent ‘Dragonfly’
  • Added rules for user-agent ‘Google-adstxt’
  • … 14 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +1 −4
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_christmas_board_reinforce_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_christmas_new_year_eve_board_reinforce_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_new_years_eve_board_reinforce_www.pinterest.com.xml
view diff → snapshot
Bloomberg bloomberg.com modified +48 −7
  • 🤖 AI Added Allow: /company for Claude-SearchBot
  • 🤖 AI Added Allow: /faq for Claude-SearchBot
  • 🤖 AI Added Allow: /latam for Claude-SearchBot
  • 🤖 AI Added Allow: /professional for Claude-SearchBot
  • 🤖 AI Added Allow: /tc for Claude-SearchBot
  • 🤖 AI Added Allow: /company for Claude-User
  • … 41 more change(s)
view diff → snapshot
Gilead Sciences gilead.com modified +1 −1
  • Added Disallow: /-/media/gileadspain/pdfs/other/gilead-sciences-spain-efpia-report-2023.pdf for *
  • Removed Disallow: /-/media/gileadspain/pdfs/about/transparency/2025/gilead-sciences-spain-efpia-methodological-note-2025.pdf for *
view diff → snapshot
Trend Micro trendmicro.com modified +3 −10
  • Removed all rules for user-agent ‘ChatGPT-User/2.0’
  • 🤖 AI Removed Allow: *.css$ for Amazonbot
  • 🤖 AI Removed Allow: *.js$ for Amazonbot
  • 🤖 AI Removed Disallow: /*/business/parsing-pages/ for Amazonbot
  • 🤖 AI Removed Disallow: /*/business/trials_parsing_pages/ for Amazonbot
  • 🤖 AI Removed Disallow: /download/* for Amazonbot
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Hulu hulu.com modified +132 −207
  • Added rules for user-agent ‘360Spider’
  • Added rules for user-agent ‘Adidxbot’
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • … 45 more change(s)
view diff → snapshot