The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2614snapshots

1442changes detected

231this week

All changes 🤖 AI-crawler changes Hiding initial archives

Sysco sysco.com modified +1 −1 24d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Arkose Labs arkoselabs.com modified +0 −2 24d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

The Independent independent.co.uk modified +1 −2 24d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Figma Community figma.com modified +1 −0 24d ago

🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for CHATGPT-User
🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for Claude-SearchBot
🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for OAI-SearchBot
🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for PerplexityBot

view diff → snapshot

Adidas adidas.com modified +1 −1 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pearson pearson.com modified +7 −0 25d ago

Added Disallow: /component-definition.json for *
Added Disallow: /component-filters.json for *
Added Disallow: /component-models.json for *
Added Disallow: /content/pearson-sites/language-masters/* for *
Added Disallow: /en-us/fragments/* for *
Added Disallow: /en-us/platinum-test/* for *
… 1 more change(s)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 25d ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +16 −16 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

South China Morning Post scmp.com modified +3 −0 25d ago

Added Disallow: /presented/better-life/career-education/topics/edtalk-harbour-school/article/3199785/rethinking-5-educational-myths-through-experiential-learning-approach for Googlebot

view diff → snapshot

Enphase Energy enphase.com modified +6 −0 25d ago

Added Allow: /core/*.avif for *
Added Allow: /core/*.webp for *
Added Allow: /profiles/*.avif for *
Added Allow: /profiles/*.webp for *
Added Disallow: /index.php/search? for *
Added Disallow: /search? for *

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 25d ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 25d ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

The Independent independent.co.uk modified +2 −1 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +6 −6 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 25d ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +17 −17 25d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 26d ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

Amgen amgen.com modified +2 −0 26d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 26d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 26d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

The Independent independent.co.uk modified +1 −2 26d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +27 −0 26d ago

Added rules for user-agent ‘*’
Added rules for user-agent ‘Mediapartners-Google’
Added sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 26d ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 26d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 26d ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot