The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5433snapshots

4298changes detected

233this week

All changes 🤖 AI-crawler changes Showing initial archives

Colombia (Presidencia) presidencia.gov.co created +0 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Roblox roblox.com modified +61 −0 2mo ago

🤖 AI Blocked Amazonbot (Amazon) entirely
🤖 AI Blocked Applebot-Extended (Apple) entirely
🤖 AI Blocked Bytespider (ByteDance) entirely
🤖 AI Blocked CCBot (Common Crawl) entirely
🤖 AI Blocked ClaudeBot (Anthropic) entirely
Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
… 4 more change(s)

view diff → snapshot

Owens Corning owenscorning.com modified +32 −1 2mo ago

Added rules for user-agent ‘*’
Added sitemap https://www.owenscorning.com/sitemap-1.xml
Added sitemap https://www.owenscorning.com/sitemap.xml

view diff → snapshot

Amgen amgen.com modified +2 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +240 −240 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Sysco sysco.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Figma Community figma.com modified +29 −0 2mo ago

🤖 AI Added Allow: /de-de/blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
🤖 AI Added Allow: /de-de/blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /de-de/blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
🤖 AI Added Allow: /es-es/blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
🤖 AI Added Allow: /es-es/blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /es-es/blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
… 45 more change(s)

view diff → snapshot

Pinterest pinterest.com modified +13 −7 2mo ago

Added sitemap https://www.pinterest.com/v3_sitemaps/qem_klp_vlm_navboost_sample_high_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/qem_klp_vlm_navboost_sample_low_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/qem_klp_vlm_navboost_sample_medium_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/qem_klp_vlm_vase_high_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/qem_klp_vlm_vase_low_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/qem_klp_vlm_vase_medium_www.pinterest.com.xml

view diff → snapshot

LinkedIn linkedin.com modified +113 −0 2mo ago

Added rules for user-agent ‘QQBot’

view diff → snapshot

The Washington Post washingtonpost.com modified +1 −0 2mo ago

Added Disallow: /partials/ for *

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Ralph Lauren ralphlauren.com modified +21 −19 2mo ago

Added Disallow: */Checkout-GetBillCountries for *
Added Disallow: */CheckoutServices-ExistingCustomer for *
Added Disallow: */Header-MenuFlyouts for *
Added Disallow: */Product-AlsoAvailable for *
Added Disallow: */Product-HitTile for *
Added Disallow: */Product-Look for *
… 26 more change(s)

view diff → snapshot

Amgen amgen.com modified +0 −2 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Gilead Sciences gilead.com modified +4 −4 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +28 −28 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +18 −18 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

De Telegraaf telegraaf.nl modified +16 −1 2mo ago

Blocked user-agent ‘Grok’ entirely
Blocked user-agent ‘Grok-DeepSearch’ entirely
Blocked user-agent ‘Grokbot’ entirely
Blocked user-agent ‘xAI-Bot’ entirely
Blocked user-agent ‘xAI-Grok’ entirely

view diff → snapshot

Adorama adorama.com modified +1 −0 2mo ago

Added Disallow: /nb/* for *

view diff → snapshot

ElevenLabs elevenlabs.io modified +1 −2 2mo ago

Removed Disallow: /docs/api/ for Twitterbot

view diff → snapshot

Avery Dennison averydennison.com modified +1 −1 2mo ago

Crawl-delay for *: 10 → 1

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +28 −28 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Gilead Sciences gilead.com modified +5 −5 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Colombia (Presidencia) presidencia.gov.co removed +0 −9 2mo ago

robots.txt is no longer served (HTTP 404)

Veolia veolia.com modified +1 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot