NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Komo komo.ai modified +5 −2
  • Added Allow: /discover for *
  • Added Allow: /share/ for *
  • Added Disallow: /auth/ for *
  • Added Disallow: /library for *
  • Added Disallow: /settings for *
  • Removed Disallow: /api/ for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Barracuda barracuda.com created +27 −27
  • Added Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for *
  • Removed Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for *
  • Added Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for DotBot
  • Removed Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for DotBot
  • Added Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for sistrix
  • Removed Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Snapchat snapchat.com modified +0 −5
  • Removed sitemap https://www.snapchat.com/sitemaps/curated_topic/sitemap-index.xml
  • Removed sitemap https://www.snapchat.com/sitemaps/ddb_topic_all/sitemap-index.xml
  • Removed sitemap https://www.snapchat.com/sitemaps/ddb_topic_eight_plus/sitemap-index.xml
  • Removed sitemap https://www.snapchat.com/sitemaps/ddb_topic_up_to_eight/sitemap-index.xml
  • Removed sitemap https://www.snapchat.com/sitemaps/extended_curated_topic/sitemap-index.xml
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Qwant qwant.com modified +6 −2
  • Added Disallow: /?*bottomMenu=* for *
  • Added Disallow: /?*drawer=* for *
  • Added Disallow: /?*home=* for *
  • Removed Disallow: /account for *
  • 🤖 AI Added Disallow: /?*bottomMenu=* for AI2Bot
  • 🤖 AI Added Disallow: /?*drawer=* for AI2Bot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −225
  • robots.txt is no longer served (HTTP 404)
Slate slate.com modified +5 −0
  • Added rules for user-agent ‘captify-crawler’
  • Added rules for user-agent ‘GnowitNewsbot’
  • Added rules for user-agent ‘peer39_crawler’
  • Blocked user-agent ‘Reflectionbot’ entirely
  • Added rules for user-agent ‘SirdataBot’
view diff → snapshot
Scale AI scale.com modified +256 −0
  • Added rules for user-agent ‘Discordbot’
  • Added rules for user-agent ‘facebookexternalhit’
  • Added rules for user-agent ‘LinkedInBot’
  • Added rules for user-agent ‘Slackbot’
  • Added rules for user-agent ‘Slackbot-LinkExpanding’
  • Added rules for user-agent ‘TelegramBot’
  • … 2 more change(s)
view diff → snapshot
Reuters reuters.com modified +1 −0
  • Added sitemap https://www.reuters.com/service/archive-sitemap/sitemap.xml
view diff → snapshot
S&P Global spglobal.com modified +8 −0
  • Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/articles-sitemap-index.xml
  • Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/entities-sitemap-index.xml
  • Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/news-sitemap.xml
  • Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/sitemap.xml
  • Added sitemap https://panama.ratings.spglobal.com/ratings/es/sitemaps/articles-sitemap-index.xml
  • Added sitemap https://panama.ratings.spglobal.com/ratings/es/sitemaps/entities-sitemap-index.xml
  • … 2 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot