- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2586snapshots
1413changes detected
240this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Added Disallow: https://metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
- Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
- … 3 more change(s)
- 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
- 🤖 AI Removed Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
- 🤖 AI Removed Allow: /fr-fr/customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for Claude-SearchBot
- 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for Claude-SearchBot
- … 10 more change(s)
- Blocked user-agent ‘AhrefsBot’ entirely
- Blocked user-agent ‘Archive.org_bot’ entirely
- Blocked user-agent ‘Brightbot 1.0’ entirely
- Blocked user-agent ‘ByteSpider Desktop’ entirely
- Blocked user-agent ‘ExaBot’ entirely
- Blocked user-agent ‘Petalbot’ entirely
- … 5 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added rules for user-agent ‘*’
- Added sitemap https://www.owenscorning.com/sitemap-1.xml
- Added sitemap https://www.owenscorning.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- 🤖 AI Added Allow: /it-it/soluzioni/assistente-handoff-design-ai/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/soluzioni/controllo-accessibilita-ai/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-figma-a-codice/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-mockup-a-prototipo/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-sketch-a-figma/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/soluzioni/denominazione-livelli-ai/$ for CHATGPT-User
- … 45 more change(s)
- Added Disallow: /business/help/search*&query= for Applebot
- Added Disallow: /business/help/search*?query= for Applebot
- Added Disallow: /business/news*&tags for Applebot
- Added Disallow: /business/news*?tags for Applebot
- Added Disallow: /business/search*&q= for Applebot
- Added Disallow: /business/search*?q= for Applebot
- … 45 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /qbr-tcs for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /investments$ for *
- Added Disallow: /investments/ for *
- Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod.adobecqms.net/* for *
- … 3 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)