The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5433snapshots

4298changes detected

233this week

All changes 🤖 AI-crawler changes Showing initial archives

Europe PMC europepmc.org modified +141 −0 2mo ago

Added rules for user-agent ‘baiduspider’
Added rules for user-agent ‘bingbot’
🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
Added rules for user-agent ‘Googlebot’
Added rules for user-agent ‘MSNbot’
🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
… 18 more change(s)

view diff → snapshot

Fox News foxnews.com modified +1 −3 2mo ago

Removed Disallow: /api/article-search for *
Removed Disallow: /xid for *
Added sitemap https://video.foxnews.com/videositemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +28 −28 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

F5 f5.com created +0 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com created +0 −78 2mo ago

Removed Disallow: /Sii8yP_qTEmmJiKsPYF7hZ21UiFM2l1ShyELGEXtZ9o=.html for *
Crawl-delay for *: 1 → none
Removed Disallow: /Sii8yP_qTEmmJiKsPYF7hZ21UiFM2l1ShyELGEXtZ9o=.html for DotBot
Crawl-delay for DotBot: 1 → none
Removed Disallow: /Sii8yP_qTEmmJiKsPYF7hZ21UiFM2l1ShyELGEXtZ9o=.html for sistrix
Crawl-delay for sistrix: 1 → none
… 3 more change(s)

view diff → snapshot

Europe PMC europepmc.org modified +0 −141 2mo ago

Removed all rules for user-agent ‘baiduspider’
Removed all rules for user-agent ‘bingbot’
🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
Removed all rules for user-agent ‘Googlebot’
Removed all rules for user-agent ‘MSNbot’
🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
… 19 more change(s)

view diff → snapshot

Amphenol amphenol.com modified +3 −0 2mo ago

Added Disallow: /Search for *

view diff → snapshot

DSV dsv.com modified +26 −26 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +2 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +90 −90 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Figma Community figma.com modified +3 −0 2mo ago

🤖 AI Added Allow: /blog/measuring-the-impact-of-ai/$ for CHATGPT-User
🤖 AI Added Allow: /blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
🤖 AI Added Allow: /webinars/expressive-design-tools/$ for CHATGPT-User
🤖 AI Added Allow: /blog/measuring-the-impact-of-ai/$ for Claude-SearchBot
🤖 AI Added Allow: /blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for Claude-SearchBot
🤖 AI Added Allow: /webinars/expressive-design-tools/$ for Claude-SearchBot
… 9 more change(s)

view diff → snapshot

Tanium tanium.com modified +10 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Europe PMC europepmc.org modified +141 −0 2mo ago

Added rules for user-agent ‘baiduspider’
Added rules for user-agent ‘bingbot’
🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
Added rules for user-agent ‘Googlebot’
Added rules for user-agent ‘MSNbot’
🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
… 18 more change(s)

view diff → snapshot

Pinterest pinterest.com modified +10 −10 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amphenol amphenol.com modified +0 −3 2mo ago

Removed Disallow: /Search for *

view diff → snapshot

Applied Materials appliedmaterials.com removed +0 −21 2mo ago

robots.txt is no longer served (HTTP 404)

Kraken kraken.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +24 −24 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +18 −18 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com removed +0 −215 2mo ago

robots.txt is no longer served (HTTP 404)

Orange orange.com modified +2 −6 2mo ago

Removed Disallow: /en/communiques/* for *
Removed Disallow: /fr/press-release/* for *

view diff → snapshot

Reuters reuters.com modified +3 −0 2mo ago

🤖 AI Blocked Applebot-Extended (Apple) entirely

view diff → snapshot

American Family Insurance amfam.com modified +3 −3 2mo ago

Added sitemap https://www.amfam.com/sitemap.xml
Removed sitemap www.amfam.com/sitemap.xml

view diff → snapshot

AutoNation autonation.com modified +3 −0 2mo ago

Added sitemap https://www.autonation.com/sitemap/srp/make_bodytype
Added sitemap https://www.autonation.com/sitemap/srp/make_model_year
Added sitemap https://www.autonation.com/sitemap/srp/make_year

view diff → snapshot