- 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler Google-Extended (Google)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- … 3 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2647snapshots
1476changes detected
231this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- Blocked user-agent ‘Diffbot-User’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- 🤖 AI Blocked Timpibot (Timpi) entirely
- 🤖 AI Blocked YouBot (You.com) entirely
- 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘Bingbot’
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- Removed all rules for user-agent ‘Diffbot-User’
- Removed all rules for user-agent ‘Scrapy’
- 🤖 AI Unblocked Timpibot (Timpi) - rules removed
- 🤖 AI Unblocked YouBot (You.com) - rules removed
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘Amzn-User’
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Blocked user-agent ‘Google-CloudVertexBot’ entirely
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*&serpstart= for *
- Added Disallow: /*/vaclk for *
- Removed Allow: /hire/*?*isid= for *
- Removed Allow: /personeel/*?*isid= for *
- Removed Allow: /reclutamiento/*?*isid= for *
- Removed Allow: /recruiting/*?*isid= for *
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added Disallow: /cdn-cgi/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)