- Removed Disallow: /App_Browsers/ for *
- Removed Disallow: /App_Data/ for *
- Removed Disallow: /App_config/ for *
- Removed Disallow: /Sitecore for *
- Removed Disallow: /sitecore modules/ for *
- Removed Disallow: /sitecore_files/ for *
- … 3 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Removed Disallow: /*/find for AmazonAdBot
- Removed Disallow: /*/find/ for AmazonAdBot
- Removed Disallow: /*/search/name/* for AmazonAdBot
- Removed Disallow: /*/search/title/* for AmazonAdBot
- Removed Disallow: /find for AmazonAdBot
- Removed Disallow: /find/ for AmazonAdBot
- … 2 more change(s)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.sofi.com/newsroom/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /*content/dam/assets/microsites/credit-suisse/documents/corporate-clients for *
- Removed Disallow: /*content/dam/assets/microsites/credit-suisse/documents/private-clients for *
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
- 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
- Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
- … 4 more change(s)
- Added sitemap https://exa.ai/docs/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /gen/ for *
- Removed Disallow: /gen/ for Googlebot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /Search for *
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- 🤖 AI Unblocked ChatGPT-User (OpenAI) - rules removed
- Added Disallow: /home-cir-2026$ for *
- Added Disallow: /serving-communities/cir-2026$ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘BrightEdge’
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)