The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2571snapshots

1398changes detected

225this week

All changes 🤖 AI-crawler changes Showing initial archives

KLA kla.com modified +1 −0 3h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +27 −27 3h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 3h ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 3h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +16 −16 3h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

X (Twitter) x.com modified +2 −2 3h ago

Added Allow: /search?q=%23 for Bingbot
Removed Disallow: /search?q= for Bingbot
Added Allow: /search?q=%23 for Googlebot
Removed Disallow: /search?q= for Googlebot

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +27 −0 9h ago

Added rules for user-agent ‘*’
Added rules for user-agent ‘Mediapartners-Google’
Added sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

Jacobs Solutions jacobs.com modified +1 −0 9h ago

Added Disallow: /user/ for *

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 9h ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +27 −27 9h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 9h ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 9h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 9h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

The Telegraph telegraph.co.uk modified +1 −1 9h ago

🤖 AI Unblocked Google-Extended (Google)
🤖 AI Added Allow: / for Google-Extended

view diff → snapshot

Figma Community figma.com modified +21 −0 9h ago

🤖 AI Added Allow: /blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
🤖 AI Added Allow: /de-de/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
🤖 AI Added Allow: /de-de/blog/what-the-design-to-code-loop-unlocks/$ for CHATGPT-User
🤖 AI Added Allow: /es-es/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
🤖 AI Added Allow: /es-es/blog/what-the-design-to-code-loop-unlocks/$ for CHATGPT-User
🤖 AI Added Allow: /es-la/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
… 45 more change(s)

view diff → snapshot

Pinterest pinterest.com modified +10 −10 9h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Cencora cencora.com modified +37 −1 9h ago

🤖 AI Added rules for AI crawler Amazonbot (Amazon)
🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
🤖 AI Added rules for AI crawler CCBot (Common Crawl)
🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
🤖 AI Added rules for AI crawler Claude-User (Anthropic)
… 7 more change(s)

view diff → snapshot

ChatGPT chatgpt.com modified +2 −0 9h ago

Added Allow: /voice$ for *
Added Allow: /voice/$ for *

view diff → snapshot

Adidas adidas.com modified +1 −1 16h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +0 −27 16h ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

Jacobs Solutions jacobs.com modified +0 −1 16h ago

Removed Disallow: /user/ for *

view diff → snapshot

DSV dsv.com modified +22 −22 16h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 16h ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 16h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +13 −13 16h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Etsy etsy.com modified +1 −1 16h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Jacobs Solutions jacobs.com modified +1 −0 22h ago

Added Disallow: /user/ for *

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 22h ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +26 −26 22h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 22h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot