NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2643snapshots
1471changes detected
226this week
All changes 🤖 AI-crawler changes Hiding initial archives
Indeed indeed.com modified +30 −177
  • 🤖 AI Added rules for AI crawler AI2Bot (Allen Institute for AI)
  • 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
  • Added rules for user-agent ‘Google-Agent’
  • Added rules for user-agent ‘Google-CloudVertexBot’
  • Added rules for user-agent ‘Grok-DeepSearch’
  • Added rules for user-agent ‘GrokBot’
  • … 17 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-website-beispiele/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/ejemplos-sitios-web-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/andrew-boz-bosworths-10-rules-for-navigating-the-next-design-paradigm/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/resource-library/ejemplos-sitios-web-de-ia/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +34 −33
  • Added sitemap https://www.pinterest.com/v3_sitemaps/image_pin_sitemap_recently_created_www.pinterest.com.xml
view diff → snapshot
LinkedIn linkedin.com modified +120 −1
  • 🤖 AI Unblocked Claude-SearchBot (Anthropic)
  • 🤖 AI Added Allow: /business/learning/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /business/sales/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /help/ for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls* for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls/retargeting-opt-out* for Claude-SearchBot
  • … 45 more change(s)
view diff → snapshot
Hilton Worldwide hilton.com modified +29 −71
  • Added rules for user-agent ‘facebookexternalhit’
  • Removed all rules for user-agent ‘Bingbot’
  • Removed all rules for user-agent ‘Googlebot’
  • Added Disallow: /attend-my-event/ for *
  • Added Disallow: /auth/ for *
  • Added Disallow: /auth2/ for *
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Deere deere.com modified +1 −13
  • Removed Disallow: /asia/en/search/ for *
  • Removed Disallow: /en/articles/ for *
  • Removed Disallow: /en/finance/offers-discounts/index.html for *
  • Removed Disallow: /en/news/all-news/ for *
  • Removed Disallow: /en/news/all-news/$ for *
  • Removed Disallow: /en/news/all-news/index.html for *
  • … 8 more change(s)
view diff → snapshot
Flickr flickr.com modified +5 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Expeditors International expeditors.com modified +14 −15
  • Added Disallow: /404-page-not-found/ for *
  • Added Disallow: /error/ for *
  • Added Disallow: /search/ for *
  • Removed Disallow: /404-page-not-found for *
  • Removed Disallow: /error for *
  • Removed Disallow: /search for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Slack slack.com modified +0 −3
  • Removed Allow: /*?*&geocode=* for *
  • Removed Allow: /*?geocode=* for *
  • Removed Disallow: /*?* for *
view diff → snapshot
Figma Community figma.com modified +6 −3
  • 🤖 AI Added Allow: /blog/how-season-5-merch-came-to-life/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • … 30 more change(s)
view diff → snapshot