NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Barracuda barracuda.com created +27 −27
  • Added Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for *
  • Removed Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for *
  • Added Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for DotBot
  • Removed Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for DotBot
  • Added Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for sistrix
  • Removed Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Toyota toyota.com modified +1 −15
  • Removed Disallow: /espanol/vehicles/ for *
  • Removed Disallow: /vehicles/ for *
  • Removed Disallow: /espanol/vehicles/ for Applebot
  • Removed Disallow: /vehicles/ for Applebot
  • 🤖 AI Removed Disallow: /espanol/vehicles/ for ChatGPT-User
  • 🤖 AI Removed Disallow: /vehicles/ for ChatGPT-User
  • … 8 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap https://www.iqvia.com/image-sitemap.xml
  • Added sitemap https://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
IGN ign.com modified +1 −1
  • Added Allow: /sponsored/ for "008"
  • Removed Allow: /sponsored/ for *
  • Added Allow: /sponsored/ for AddSearchBot
  • 🤖 AI Added Allow: /sponsored/ for AI2Bot
  • Added Allow: /sponsored/ for Ai2Bot-DeepResearchEval
  • Added Allow: /sponsored/ for Ai2Bot-Dolma
  • … 45 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −225
  • robots.txt is no longer served (HTTP 404)
Slate slate.com modified +18 −0
  • Blocked user-agent ‘ApifyBot’ entirely
  • Blocked user-agent ‘ApifyWebsiteContentCrawler’ entirely
  • Blocked user-agent ‘Bravebot’ entirely
  • Blocked user-agent ‘Brightbot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘Crawlspace’ entirely
  • … 12 more change(s)
view diff → snapshot
Threads threads.net modified +28 −0
  • Added rules for user-agent ‘deepcrawl’
  • Added rules for user-agent ‘lumar’
view diff → snapshot
Instagram instagram.com modified +22 −0
  • Added rules for user-agent ‘deepcrawl’
  • Added rules for user-agent ‘lumar’
view diff → snapshot
Facebook facebook.com modified +98 −0
  • Blocked user-agent ‘deepcrawl’ entirely
  • Blocked user-agent ‘lumar’ entirely
view diff → snapshot
Tidal tidal.com initial
  • First snapshot of robots.txt archived
snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Marsh McLennan marshmclennan.com modified +22 −4
  • Added Disallow: *.cookiepolicy.json* for *
  • Added Disallow: /*.html/content/ for *
  • Added Disallow: /*/_jcr_content/ for *
  • Added Disallow: /*/campaigns.html for *
  • Added Disallow: /*/confirmation.html for *
  • Added Disallow: /*/designs/ for *
  • … 17 more change(s)
view diff → snapshot
Heidelberg Materials heidelbergmaterials.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Yandex yandex.ru modified +12 −0
  • Added Disallow: /realty/callme for *
  • Added Disallow: /realty/map_dust for *
  • Added Disallow: /realty/map_pins for *
  • Added Disallow: /realty/result for *
  • Added Disallow: /realty/callme for Bingbot
  • Added Disallow: /realty/map_dust for Bingbot
  • … 6 more change(s)
view diff → snapshot
Petrobras petrobras.com.br modified +6 −6
  • Added Allow: /documents/d/ for *
  • Added Allow: /o/tema-site-externo-petrobras/images/ for *
  • Added Disallow: /*/group/ for *
  • Removed Allow: /documents/d/nossa-energia/ for *
  • Removed Disallow: /c/portal/ for *
  • Removed Disallow: /documents/d/* for *
  • … 1 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot