- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
- 🤖 AI Added rules for AI crawler Google-Extended (Google)
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Added Disallow: /marketplace/az/* for *
- Added Disallow: /marketplace/ca/* for *
- Added Disallow: /marketplace/fl/* for *
- Added Disallow: /marketplace/ga/* for *
- Added Disallow: /marketplace/ia/* for *
- … 23 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: (empty) for *
- Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Removed Disallow: /marketplace/az/* for *
- Removed Disallow: /marketplace/ca/* for *
- Removed Disallow: /marketplace/fl/* for *
- Removed Disallow: /marketplace/ga/* for *
- … 23 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘__usage_ai-training’ entirely
- Added rules for user-agent ‘*’
- 🤖 AI Blocked GPTBot (OpenAI) entirely
- Added sitemap https://www.universalmusic.com/sitemap_index.xml
- Blocked user-agent ‘*’ entirely
- Removed Disallow: /br/pt/cm.html for *
- Removed Disallow: /br/pt/cm/ for *
- Removed Disallow: /cn/zh/cm.html for *
- Removed Disallow: /cn/zh/cm/ for *
- Removed Disallow: /content/dam/www/assets/partners/us/en/mississippi/ for *
- … 45 more change(s)
- Added rules for user-agent ‘*’
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked CCBot (Common Crawl) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- 🤖 AI Blocked Google-Extended (Google) entirely
- … 4 more change(s)
- 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
- Removed all rules for user-agent ‘Applebot’
- Removed all rules for user-agent ‘AwarioRssBot’
- Removed all rules for user-agent ‘AwarioSmartBot’
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
- … 17 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for CHATGPT-User
- 🤖 AI Added Allow: /pt-br/legal/privacy/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for Claude-SearchBot
- 🤖 AI Added Allow: /pt-br/legal/privacy/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for Claude-User
- 🤖 AI Added Allow: /pt-br/legal/privacy/$ for Claude-User
- … 4 more change(s)