- Added sitemap https://www.walmart.com/sitemap_product_03.xml
- Removed sitemap https://www.walmart.com/sitemap_itp_03.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added Disallow: /dcp/api/* for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Added Disallow: https://metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
- Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
- … 3 more change(s)
- Added Disallow: /buehne/ for *
- Added Disallow: /sonstiges/intern/ for *
- Added Disallow: /vorprod/ for *
- Added Disallow: /vorproduktion/ for *
- Added sitemap https://www.ucl.ac.uk/bloomsbury-theatre/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/centre-law-europe/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/2034/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/3dgeoinfo/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/about/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/academic-manual/sitemap.xml
- … 45 more change(s)
- Unblocked user-agent ‘*’
- Added Disallow: /search for *
- Added sitemap https://www.linde.com/sitemap.xml
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /investments$ for *
- Added Disallow: /investments/ for *
- Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod.adobecqms.net/* for *
- … 3 more change(s)
- Blocked user-agent ‘*’ entirely
- Removed Disallow: /search for *
- Removed sitemap https://www.linde.com/sitemap.xml
- Added Allow: /llms.txt for *
- Added Disallow: /locations for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Removed Allow: /core/*.avif for *
- Removed Allow: /core/*.webp for *
- Removed Allow: /profiles/*.avif for *
- Removed Allow: /profiles/*.webp for *
- Removed Disallow: /index.php/search? for *
- Removed Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.imperva.com/vs/sitemap_index.xml
- Added rules for user-agent ‘*’
- Added sitemap https://www.diffbot.com/sitemap.xml