- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘ChatGPT-User/2.0’
- 🤖 AI Removed Allow: *.css$ for Amazonbot
- 🤖 AI Removed Allow: *.js$ for Amazonbot
- 🤖 AI Removed Disallow: /*/business/parsing-pages/ for Amazonbot
- 🤖 AI Removed Disallow: /*/business/trials_parsing_pages/ for Amazonbot
- 🤖 AI Removed Disallow: /download/* for Amazonbot
- … 45 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘360Spider’
- Added rules for user-agent ‘Adidxbot’
- Added rules for user-agent ‘AdsBot-Google’
- Added rules for user-agent ‘AdsBot-Google-Mobile’
- Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
- 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
- … 45 more change(s)
- Removed Disallow: /apply/credit-cards/discover/preapprove/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘CohereBot’ entirely
- Blocked user-agent ‘CragCrawler’ entirely
- Blocked user-agent ‘GeistHaus-PageFetcher’ entirely
- Blocked user-agent ‘Kimi-SearchBot’ entirely
- Blocked user-agent ‘Kimi-User’ entirely
- Blocked user-agent ‘KimiBot’ entirely
- … 3 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /store/XboxComMsCom3PAdsOptOutCookieSync.html for *
- Added Disallow: /store/XboxComMsComCartMuidSync.html for *
- Added Disallow: /account/api/ for *
- Added Disallow: /auth/api/ for *
- Added Disallow: /demo-drive/api/ for *
- Added Disallow: /experience/api/ for *
- Added Disallow: /quad/api/ for *
- Added Disallow: /root/api/ for *
- … 4 more change(s)
- Added Disallow: /index.php/admin/ for *
- Added Disallow: /index.php/comment/reply/ for *
- Added Disallow: /index.php/filter/tips for *
- Added Disallow: /index.php/node/add/ for *
- Added Disallow: /index.php/user/ for *
- Removed Allow: /content/dam/ for *
- … 37 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://www.kraken.com/sitemaps/embed-convert-buy.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /news/the-harry-trial/* for *
- Formatting/comment-only change (no semantic difference)