The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5420snapshots

4284changes detected

241this week

All changes 🤖 AI-crawler changes Showing initial archives

DSV dsv.com modified +24 −24 10d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Arrow Electronics arrow.com modified +1 −3 10d ago

🤖 AI Unblocked GPTBot (OpenAI)
🤖 AI Added Allow: / for GPTBot
🤖 AI Removed Allow: /ais/* for GPTBot
🤖 AI Removed Allow: /globalecs/* for GPTBot

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 10d ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 10d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +13 −13 10d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Vimeo vimeo.com modified +340 −0 10d ago

Added rules for user-agent ‘ia_archiver’
Added Disallow: /api/v3/*/comments for *
Added Disallow: /api/v3/me$ for *
Added Disallow: /api/v3/me/ for *
Added Disallow: /api/v3/me? for *
Added Disallow: /api/v3/modworld for *

view diff → snapshot

Mastodon mastodon.social modified +21 −1 10d ago

🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
Blocked user-agent ‘Ai2Bot-Dolma’ entirely
🤖 AI Blocked anthropic-ai (Anthropic) entirely
🤖 AI Blocked Applebot-Extended (Apple) entirely
🤖 AI Blocked Bytespider (ByteDance) entirely
🤖 AI Blocked CCBot (Common Crawl) entirely
… 14 more change(s)

view diff → snapshot

Adorama adorama.com modified +0 −1 10d ago

Removed Allow: *Filter-By_Specials for *

view diff → snapshot

Etsy etsy.com modified +1 −1 10d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +140 −137 10d ago

Added sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_priority_segment_2026q4_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/priority_segment_2026q4_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/shopping_sitemap_www.pinterest.com.xml

view diff → snapshot

Stanford University stanford.edu modified +1 −0 10d ago

Added Disallow: /statuscake for *

view diff → snapshot

Adidas adidas.com modified +1 −1 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Constellation Brands cbrands.com modified +1 −0 11d ago

Added Disallow: /.well-known/shopify/monorail for *

view diff → snapshot

Regeneron Pharmaceuticals regeneron.com created +0 −0 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 11d ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Infineon infineon.com modified +1 −0 11d ago

Added Disallow: */secure* for *

view diff → snapshot

WordPress wordpress.com modified +1 −1 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Regeneron Pharmaceuticals regeneron.com removed +0 −98 11d ago

robots.txt is no longer served (HTTP 404)

KLA kla.com modified +1 −0 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +27 −27 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Europe PMC europepmc.org modified +70 −1 11d ago

🤖 AI Blocked anthropic-ai (Anthropic) entirely
Blocked user-agent ‘atlassian-bot’ entirely
Blocked user-agent ‘AwarioRssBot’ entirely
Blocked user-agent ‘AwarioSmartBot’ entirely
Blocked user-agent ‘Baiduspider’ entirely
Blocked user-agent ‘BorderxBot’ entirely
… 19 more change(s)

view diff → snapshot

HarperCollins harpercollins.com modified +1 −0 11d ago

Added Disallow: /.well-known/shopify/monorail for *

view diff → snapshot

Kohl's kohls.com modified +0 −3 11d ago

Removed Disallow: /*&bvstate= for *
Removed Disallow: /*?bvstate= for *
Removed Disallow: /bvreviews/ for *

view diff → snapshot

Decathlon decathlon.com modified +1 −0 11d ago

Added Disallow: /.well-known/shopify/monorail for *

view diff → snapshot

DSV dsv.com modified +22 −22 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +12 −12 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 11d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot