- Added sitemap https://dotcom.ecolab.com/sitemap.xml
- Removed sitemap https://www.ecolab.com/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘AhrefsSiteAudit’ entirely
- 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘audisto’ entirely
- Blocked user-agent ‘audisto-essential’ entirely
- Blocked user-agent ‘barkrowler’ entirely
- … 45 more change(s)
- Crawl-delay for *: none → 10
- Added sitemap http://webz.io/sitemap_index.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Added Allow: /community/gallery$ for *
- Added Allow: /community/gallery/assets$ for *
- Added Allow: /community/gallery/challenges$ for *
- Added Disallow: /community/gallery for *
- Removed Disallow: /community/gallery/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked CCBot (Common Crawl) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
- … 4 more change(s)
- Added rules for user-agent ‘*’
- Added sitemap https://www.owenscorning.com/sitemap-1.xml
- Added sitemap https://www.owenscorning.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)