- Added sitemap https://www.averydennison.com/sitemap-main.xml
- Removed sitemap https://www.averydennison.com/en/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2634snapshots
1462changes detected
251this week
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-bg-bg-sitemap_index.xml
- Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ca-es-sitemap_index.xml
- Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-cs-cz-sitemap_index.xml
- Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-da-dk-sitemap_index.xml
- Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-de-de-sitemap_index.xml
- Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-ae-sitemap_index.xml
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /native/business/banking-finance/topics/wealth-across-borders/article/3357488/retired-businessman-and-lifelong-sailor-trust-anchors-process-building-wealth for Googlebot
- Added Disallow: /print/native/business/banking-finance/topics/wealth-across-borders/article/3357488/retired-businessman-and-lifelong-sailor-trust-anchors-process-building-wealth for Googlebot
- Added Disallow: /print/presented/better-life/career-education/topics/edtalk-harbour-school/article/3199785/rethinking-5-educational-myths-through-experiential-learning-approach for Googlebot
- Added Disallow: /*/help/*&query= for *
- Added Disallow: /*/help/*?query= for *
- Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for *
- Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
- Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for DotBot
- Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
- Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for sistrix
- Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for sistrix
- … 2 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /pd/*/af$ for *
- Added Disallow: /pd/*/af=$ for *
- Added Disallow: /pd/*/ar$ for *
- Added Disallow: /pd/*/az$ for *
- Added Disallow: /pd/*/be$ for *
- Added Disallow: /pd/*/bg$ for *
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Removed Disallow: /ReviewExpress- for *
- Removed Disallow: /ReviewExpress- for applebot
- Removed Disallow: /ReviewExpress- for bingbot
- 🤖 AI Removed Disallow: /ReviewExpress- for ChatGPT-User
- Removed Disallow: /ReviewExpress- for ChatGPT-User/2.0
- Removed Disallow: /ReviewExpress- for Gemini-Deep-Research
- … 3 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /my-trips/search?* for *
- Added Disallow: /mytrips/ for *
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- First snapshot of robots.txt archived
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap https://www.intuit.com/enterprise/blog/sitemap.xml
- Formatting/comment-only change (no semantic difference)