- Added Disallow: */wp-content/uploads/sites/*/documents/ for *
- Removed Disallow: /ja-jp/wp-content/uploads/sites/13/documents/ for *
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Added sitemap https://www.kraken.com/sitemaps/prices-asset-new-new-new.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
- Added rules for user-agent ‘AndiBot’
- Added rules for user-agent ‘Applebot’
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘Bingbot’
- 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
- … 17 more change(s)
- Blocked user-agent ‘Googlebot-News’ entirely
- Added sitemap https://abcnews.go.com/xmap
- Added sitemap https://abcnews.go.com/xmlLatestStories
- Added sitemap https://abcnews.go.com/xmlLatestVideos
- Removed sitemap https://abcnews.com/xmap
- Removed sitemap https://abcnews.com/xmlLatestStories
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
- Added rules for user-agent ‘Applebot’
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘BingBot’
- 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
- Added rules for user-agent ‘facebookexternalhit’
- … 43 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Crawl-delay for *: 10 → none
- Blocked user-agent ‘IndeedBot’ entirely
- Blocked user-agent ‘Indeedshopbot’ entirely
- Removed Allow: / for *
- Removed all rules for user-agent ‘DuckDuckBot’
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)