- Added sitemap https://www.sap.com/sitemap-documents-index.xml
- Added sitemap https://www.sap.com/sitemap-videos-index.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
- 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
- Added rules for user-agent ‘Applebot’
- Added rules for user-agent ‘Bingbot’
- 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
- 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
- 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
- … 6 more change(s)
- Removed Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/products/shell-panolin/about.html for *
- Removed Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/products/shell-panolin/about/ for *
- Added sitemap https://github.com/sitemap_marketing.xml
- Added sitemap https://www.gehealthcare.com/latam/es.sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sites/default/files/white-papers/ai-governance-report-germany.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-de-wp-ai-governance-report.pdf for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /sitemap.xml for *
- Added sitemap https://www.tesla.com/sitemap.xml
- Added Allow: /cn/docs/bdk for *
- Blocked user-agent ‘ExaSearchBot’ entirely
- Blocked user-agent ‘ImageBot/1.0’ entirely
- Blocked user-agent ‘TaraGroup Intelligent Bot’ entirely
- Removed Disallow: /bentoapi/ for *
- Removed sitemap https://www.commscope.com/networktype-copperbroadbandaccesssitemap.xml
- Removed sitemap https://www.commscope.com/networktype-fttxaccesssitemap.xml
- Removed sitemap https://www.commscope.com/networktype-hfcaccesssitemap.xml
- Unblocked user-agent ‘*’
- Added Disallow: */product-inquiry for *
- Added Disallow: */search-results for *
- Added Disallow: /content/dam/eastman/corporate/en/literature/private/ for *
- Added sitemap https://www.entergy.com/wp-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /nu/cartao/nu-limite-garantido for *
- Added Disallow: /nu/cartao/nu-limite-garantido for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sites/default/files/misc/healthcare-cybercrime-update-q2-2026-pt-br.pdf for *
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked anthropic-ai (Anthropic) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘atlassian-bot’ entirely
- Blocked user-agent ‘AwarioRssBot’ entirely
- Blocked user-agent ‘AwarioSmartBot’ entirely
- … 27 more change(s)
- Removed Disallow: /content/experience-fragments/ for *