NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Fortra fortra.com modified +45 −0
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 5 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Indeed indeed.com modified +30 −177
  • 🤖 AI Added rules for AI crawler AI2Bot (Allen Institute for AI)
  • 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
  • Added rules for user-agent ‘Google-Agent’
  • Added rules for user-agent ‘Google-CloudVertexBot’
  • Added rules for user-agent ‘Grok-DeepSearch’
  • Added rules for user-agent ‘GrokBot’
  • … 17 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-website-beispiele/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/ejemplos-sitios-web-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/andrew-boz-bosworths-10-rules-for-navigating-the-next-design-paradigm/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/resource-library/ejemplos-sitios-web-de-ia/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +34 −33
  • Added sitemap https://www.pinterest.com/v3_sitemaps/image_pin_sitemap_recently_created_www.pinterest.com.xml
view diff → snapshot
LinkedIn linkedin.com modified +120 −1
  • 🤖 AI Unblocked Claude-SearchBot (Anthropic)
  • 🤖 AI Added Allow: /business/learning/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /business/sales/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /help/ for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls* for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls/retargeting-opt-out* for Claude-SearchBot
  • … 45 more change(s)
view diff → snapshot
Hilton Worldwide hilton.com modified +29 −71
  • Added rules for user-agent ‘facebookexternalhit’
  • Removed all rules for user-agent ‘Bingbot’
  • Removed all rules for user-agent ‘Googlebot’
  • Added Disallow: /attend-my-event/ for *
  • Added Disallow: /auth/ for *
  • Added Disallow: /auth2/ for *
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Deere deere.com modified +1 −13
  • Removed Disallow: /asia/en/search/ for *
  • Removed Disallow: /en/articles/ for *
  • Removed Disallow: /en/finance/offers-discounts/index.html for *
  • Removed Disallow: /en/news/all-news/ for *
  • Removed Disallow: /en/news/all-news/$ for *
  • Removed Disallow: /en/news/all-news/index.html for *
  • … 8 more change(s)
view diff → snapshot