NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/botim-ai-ready-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skills-for-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for Claude-SearchBot
  • … 10 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
  • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
  • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
  • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
  • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for sistrix
  • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-us-eb-email-security-blind-spots.pdf for *
view diff → snapshot
X (Twitter) x.com modified +3 −3
  • Added Disallow: /search?q= for Bingbot
  • Removed Allow: /search?q=%23 for Bingbot
  • Added Disallow: /search?q= for facebookexternalhit
  • Removed Allow: /search?q=%23 for facebookexternalhit
  • Added Disallow: /search?q= for Googlebot
  • Removed Allow: /search?q=%23 for Googlebot
view diff → snapshot
MongoDB mongodb.com modified +2 −1
  • Added Disallow: /company/blog/search for *
  • Added Disallow: /docs/site-search/ for *
  • Removed Disallow: /docs/search/ for *
view diff → snapshot
Huntington Ingalls Industries hii.com modified +70 −3
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 9 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Slate slate.com modified +20 −0
  • Added rules for user-agent ‘AdmixerBot’
  • Added rules for user-agent ‘AffsignalCrawler’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘CohereBot’ entirely
  • Added rules for user-agent ‘Dragonfly’
  • Added rules for user-agent ‘Google-adstxt’
  • … 14 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +1 −4
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_christmas_board_reinforce_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_christmas_new_year_eve_board_reinforce_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_new_years_eve_board_reinforce_www.pinterest.com.xml
view diff → snapshot
Bloomberg bloomberg.com modified +48 −7
  • 🤖 AI Added Allow: /company for Claude-SearchBot
  • 🤖 AI Added Allow: /faq for Claude-SearchBot
  • 🤖 AI Added Allow: /latam for Claude-SearchBot
  • 🤖 AI Added Allow: /professional for Claude-SearchBot
  • 🤖 AI Added Allow: /tc for Claude-SearchBot
  • 🤖 AI Added Allow: /company for Claude-User
  • … 41 more change(s)
view diff → snapshot
Gilead Sciences gilead.com modified +1 −1
  • Added Disallow: /-/media/gileadspain/pdfs/other/gilead-sciences-spain-efpia-report-2023.pdf for *
  • Removed Disallow: /-/media/gileadspain/pdfs/about/transparency/2025/gilead-sciences-spain-efpia-methodological-note-2025.pdf for *
view diff → snapshot