The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2667snapshots

1497changes detected

225this week

All changes 🤖 AI-crawler changes Showing initial archives

Hulu hulu.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Disneyplus disneyplus.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Disney disney.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Realtor realtor.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Zillow zillow.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Tripadvisor tripadvisor.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

WordPress wordpress.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Chase chase.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Claude claude.ai initial 1mo ago

First snapshot of robots.txt archived

snapshot

Gemini gemini.google.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Craigslist craigslist.org initial 1mo ago

First snapshot of robots.txt archived

snapshot

Indeed indeed.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Canva canva.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

ESPN espn.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Msn msn.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Roblox roblox.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Yahoo yahoo.co.jp initial 1mo ago

First snapshot of robots.txt archived

snapshot

Globo globo.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Outlook outlook.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Weather weather.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

ChatGPT chatgpt.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Baker Hughes bakerhughes.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Nippon Telegraph and Telephone group.ntt initial 1mo ago

First snapshot of robots.txt archived

snapshot

IQVIA Holdings iqvia.com modified +2 −2 1mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +16 −16 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Etsy etsy.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

SoundCloud soundcloud.com modified +29 −60 1mo ago

Blocked user-agent ‘Brightbot’ entirely
🤖 AI Blocked omgili (Webz.io) entirely
Blocked user-agent ‘omgilibot’ entirely
🤖 AI Unblocked Amazonbot (Amazon) - rules removed
🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
🤖 AI Unblocked Bytespider (ByteDance) - rules removed
… 45 more change(s)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 1mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot