- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Added Disallow: /dcp/api/* for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Added Disallow: https://metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
- Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
- … 3 more change(s)
- Added Disallow: /buehne/ for *
- Added Disallow: /sonstiges/intern/ for *
- Added Disallow: /vorprod/ for *
- Added Disallow: /vorproduktion/ for *
- Added sitemap https://www.ucl.ac.uk/bloomsbury-theatre/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/centre-law-europe/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/2034/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/3dgeoinfo/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/about/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/cms-archive/academic-manual/sitemap.xml
- … 45 more change(s)
- Unblocked user-agent ‘*’
- Added Disallow: /search for *
- Added sitemap https://www.linde.com/sitemap.xml
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /investments$ for *
- Added Disallow: /investments/ for *
- Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod.adobecqms.net/* for *
- … 3 more change(s)
- Blocked user-agent ‘*’ entirely
- Removed Disallow: /search for *
- Removed sitemap https://www.linde.com/sitemap.xml
- Added Allow: /llms.txt for *
- Added Disallow: /locations for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Removed Allow: /core/*.avif for *
- Removed Allow: /core/*.webp for *
- Removed Allow: /profiles/*.avif for *
- Removed Allow: /profiles/*.webp for *
- Removed Disallow: /index.php/search? for *
- Removed Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.imperva.com/vs/sitemap_index.xml
- Added rules for user-agent ‘*’
- Added sitemap https://www.diffbot.com/sitemap.xml
- 🤖 AI Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/customers/duotone-saves-hours/$ for CHATGPT-User
- … 22 more change(s)