NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5439snapshots
4304changes detected
239this week
All changes 🤖 AI-crawler changes Showing initial archives
WordPress wordpress.com initial
  • First snapshot of robots.txt archived
snapshot
Chase chase.com initial
  • First snapshot of robots.txt archived
snapshot
Claude claude.ai initial
  • First snapshot of robots.txt archived
snapshot
Gemini gemini.google.com initial
  • First snapshot of robots.txt archived
snapshot
Craigslist craigslist.org initial
  • First snapshot of robots.txt archived
snapshot
Indeed indeed.com initial
  • First snapshot of robots.txt archived
snapshot
Canva canva.com initial
  • First snapshot of robots.txt archived
snapshot
ESPN espn.com initial
  • First snapshot of robots.txt archived
snapshot
Msn msn.com initial
  • First snapshot of robots.txt archived
snapshot
Roblox roblox.com initial
  • First snapshot of robots.txt archived
snapshot
Yahoo yahoo.co.jp initial
  • First snapshot of robots.txt archived
snapshot
Globo globo.com initial
  • First snapshot of robots.txt archived
snapshot
Outlook outlook.com initial
  • First snapshot of robots.txt archived
snapshot
Weather weather.com initial
  • First snapshot of robots.txt archived
snapshot
ChatGPT chatgpt.com initial
  • First snapshot of robots.txt archived
snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Perplexity perplexity.ai modified +2 −0
  • Added Disallow: /marketing/prerelease/ for *
  • Added Disallow: /marketing/prerelease/ for Googlebot
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot