NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
IGN ign.com modified +1 −1
  • Added Allow: /sponsored/ for "008"
  • Removed Allow: /sponsored/ for *
  • Added Allow: /sponsored/ for AddSearchBot
  • 🤖 AI Added Allow: /sponsored/ for AI2Bot
  • Added Allow: /sponsored/ for Ai2Bot-DeepResearchEval
  • Added Allow: /sponsored/ for Ai2Bot-Dolma
  • … 45 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −225
  • robots.txt is no longer served (HTTP 404)
Slate slate.com modified +18 −0
  • Blocked user-agent ‘ApifyBot’ entirely
  • Blocked user-agent ‘ApifyWebsiteContentCrawler’ entirely
  • Blocked user-agent ‘Bravebot’ entirely
  • Blocked user-agent ‘Brightbot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘Crawlspace’ entirely
  • … 12 more change(s)
view diff → snapshot
Threads threads.net modified +28 −0
  • Added rules for user-agent ‘deepcrawl’
  • Added rules for user-agent ‘lumar’
view diff → snapshot
Instagram instagram.com modified +22 −0
  • Added rules for user-agent ‘deepcrawl’
  • Added rules for user-agent ‘lumar’
view diff → snapshot
Facebook facebook.com modified +98 −0
  • Blocked user-agent ‘deepcrawl’ entirely
  • Blocked user-agent ‘lumar’ entirely
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Marsh McLennan marshmclennan.com modified +22 −4
  • Added Disallow: *.cookiepolicy.json* for *
  • Added Disallow: /*.html/content/ for *
  • Added Disallow: /*/_jcr_content/ for *
  • Added Disallow: /*/campaigns.html for *
  • Added Disallow: /*/confirmation.html for *
  • Added Disallow: /*/designs/ for *
  • … 17 more change(s)
view diff → snapshot
Heidelberg Materials heidelbergmaterials.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Yandex yandex.ru modified +12 −0
  • Added Disallow: /realty/callme for *
  • Added Disallow: /realty/map_dust for *
  • Added Disallow: /realty/map_pins for *
  • Added Disallow: /realty/result for *
  • Added Disallow: /realty/callme for Bingbot
  • Added Disallow: /realty/map_dust for Bingbot
  • … 6 more change(s)
view diff → snapshot
Petrobras petrobras.com.br modified +6 −6
  • Added Allow: /documents/d/ for *
  • Added Allow: /o/tema-site-externo-petrobras/images/ for *
  • Added Disallow: /*/group/ for *
  • Removed Allow: /documents/d/nossa-energia/ for *
  • Removed Disallow: /c/portal/ for *
  • Removed Disallow: /documents/d/* for *
  • … 1 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Komo komo.ai modified +5 −2
  • Added Allow: /discover for *
  • Added Allow: /share/ for *
  • Added Disallow: /auth/ for *
  • Added Disallow: /library for *
  • Added Disallow: /settings for *
  • Removed Disallow: /api/ for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot