- Added rules for user-agent ‘Prerender’
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Added sitemap https://runway.com/sitemap.xml
- Removed sitemap https://runwayml.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /account/subscribe/payment for AASA-Bot
- Added Disallow: /jp/ for AASA-Bot
- Removed Disallow: /account/subscribe/payment/ for AASA-Bot
- Added Disallow: /account/subscribe/payment for ADmantX
- Added Disallow: /jp/ for ADmantX
- Removed Disallow: /account/subscribe/payment/ for ADmantX
- … 45 more change(s)
- Added rules for user-agent ‘IllumaBot/1.0’
- Added rules for user-agent ‘Publication-Access-for-Facebook’
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Added Disallow: /*j_security_check for *
- Added Disallow: /content/dam/ for *
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /lu-de/home/global-forwarding/help-center/claims-complaints-compliments-form.html for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /content/dam/abbvie-com2/pdfs/joinus for *
- Added Disallow: /content/dam/abbvie-com2/pdfs/joinus/2025-total-rewards-brochure-aug2025.pdf for *
- Added Disallow: /content/dam/abbvie-com2/pdfs/joinus/2026-abv-total-rewards-resources-guide-jan-publish.pdf for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Added Disallow: https://metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
- Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
- … 3 more change(s)
- Removed sitemap https://webz.io/sitemap_index.xml
- Added Disallow: /myft for *
- Removed Disallow: /myft/ for *
- Removed all rules for user-agent ‘Prerender’
- Blocked user-agent ‘ahrefsbot’ entirely
- Blocked user-agent ‘cazoodlebot’ entirely
- Blocked user-agent ‘dotbot/1.0’ entirely
- Blocked user-agent ‘gigabot’ entirely
- Blocked user-agent ‘mj12bot’ entirely
- Blocked user-agent ‘sogou inst spider’ entirely
- … 20 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /investments$ for *
- Added Disallow: /investments/ for *
- Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod.adobecqms.net/* for *
- … 3 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘Prerender’
- Removed all rules for user-agent ‘ahrefsbot’
- Removed all rules for user-agent ‘cazoodlebot’
- Removed all rules for user-agent ‘dotbot/1.0’
- Removed all rules for user-agent ‘gigabot’
- Removed all rules for user-agent ‘mj12bot’
- Removed all rules for user-agent ‘sogou inst spider’
- … 20 more change(s)
- Blocked user-agent ‘PetalBot’ entirely
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)