- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Added Disallow: https://metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
- Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
- … 3 more change(s)
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for Claude-SearchBot
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for OAI-SearchBot
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for PerplexityBot
- 🤖 AI Added Allow: /storyworks for ChatGPT-User
- 🤖 AI Added Allow: /storyworks/ for ChatGPT-User
- 🤖 AI Added Allow: /storyworks for Google-Extended
- 🤖 AI Added Allow: /storyworks/ for Google-Extended
- 🤖 AI Added Allow: /storyworks for GPTBot
- 🤖 AI Added Allow: /storyworks/ for GPTBot
- Added Allow: /*?*utm_medium=syndication for *
- Added Allow: /*?*utm_medium=syndication for *
- Added sitemap https://www2.totalenergies.com/sitemap.xml
- Removed sitemap https://totalenergies.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘AmazonProductDiscovery’ entirely
- Added Disallow: /*&tx_condownloadcenter_api* for *
- Added Disallow: /*&tx_conevents_list* for *
- Added Disallow: /*&tx_conrssfeed_list* for *
- Added Disallow: /*&tx_consecuredarea_* for *
- Added Disallow: /*&tx_solr* for *
- … 5 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /investments$ for *
- Added Disallow: /investments/ for *
- Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod.adobecqms.net/* for *
- … 3 more change(s)
- Removed Disallow: /search/* for *
- Removed Disallow: /search?q=* for *
- Added sitemap https://time.com/video-landing-sitemap.xml
- Added rules for user-agent ‘*’
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked CCBot (Common Crawl) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- … 5 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /can-wehearyou for *
- Added Disallow: /wehearyou for *
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /advice/search/ for *
- Added Disallow: /guides/search/ for *
- Added Disallow: /living/search/ for *
- Added Disallow: /news/search/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)