- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Blocked user-agent ‘Brightbot’ entirely
- 🤖 AI Blocked omgili (Webz.io) entirely
- Blocked user-agent ‘omgilibot’ entirely
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://www.molinahealthcare.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sinais for *
- Added Disallow: /sinais for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /application/* for *
- Added Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/page-refresh.*.js for *
- Added Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/save-article.*.js for *
- Added Disallow: /ixale/ for *
- Removed Disallow: */application/* for *
- Removed Disallow: */ixale/ for *
- … 45 more change(s)
- Added Disallow: /sites/default/files/solution-briefs/pfpt-jp-sb-active-exploits-protection.pdf for *
- Added Disallow: /motorist/oils-lubricants/helix-for-cars/our-carbon-neutral-aspiration/offsetting-remaining-emissions.html for *
- Added Disallow: /motorist/oils-lubricants/helix-for-cars/our-carbon-neutral-aspiration/offsetting-remaining-emissions/ for *
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
- 🤖 AI Blocked Claude-Web (Anthropic) entirely
- 🤖 AI Blocked cohere-ai (Cohere) entirely
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.molinahealthcare.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sites/default/files/white-papers/pfpt-au-wp-definitive-guide-shadow-ai.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-br-wp-definitive-guide-shadow-ai.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-de-wp-definitive-guide-shadow-ai.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-es-wp-definitive-guide-shadow-ai.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-fr-wp-definitive-guide-shadow-ai.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-it-wp-definitive-guide-shadow-ai.pdf for *
- … 2 more change(s)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://www.molinahealthcare.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)