- Added sitemap https://www.imperva.com/vs/sitemap_index.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
- Added rules for user-agent ‘*’
- Added sitemap https://www.diffbot.com/sitemap.xml
- 🤖 AI Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /it-it/customers/duotone-saves-hours/$ for CHATGPT-User
- … 22 more change(s)
- Added rules for user-agent ‘AmazonAdBot’
- Formatting/comment-only change (no semantic difference)
- Added Allow: /plugins for *
- Added sitemap https://totalenergies.com/sitemap.xml
- Removed sitemap https://www2.totalenergies.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.cbre.com/tools/office-space-calculator/sitemap.xml
- First snapshot of robots.txt archived
- 🤖 AI Blocked meta-externalagent (Meta) entirely
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /Games for *
- Added Disallow: /OwnerResponseApi for *
- Removed Disallow: /DiscountGroup for *
- Removed Disallow: /ExitWindow for *
- Removed Disallow: /ExternalExitWindow for *
- Removed Disallow: /OwnerResponse for *
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Added Disallow: https://metlife-prod.adobecqms.net/* for *
- Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
- Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
- … 3 more change(s)
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for CHATGPT-User
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for Claude-SearchBot
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for OAI-SearchBot
- 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for PerplexityBot
- 🤖 AI Added Allow: /storyworks for ChatGPT-User
- 🤖 AI Added Allow: /storyworks/ for ChatGPT-User
- 🤖 AI Added Allow: /storyworks for Google-Extended
- 🤖 AI Added Allow: /storyworks/ for Google-Extended
- 🤖 AI Added Allow: /storyworks for GPTBot
- 🤖 AI Added Allow: /storyworks/ for GPTBot
- Added Allow: /*?*utm_medium=syndication for *
- Added Allow: /*?*utm_medium=syndication for *
- Added sitemap https://www2.totalenergies.com/sitemap.xml
- Removed sitemap https://totalenergies.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘AmazonProductDiscovery’ entirely
- Added Disallow: /*&tx_condownloadcenter_api* for *
- Added Disallow: /*&tx_conevents_list* for *
- Added Disallow: /*&tx_conrssfeed_list* for *
- Added Disallow: /*&tx_consecuredarea_* for *
- Added Disallow: /*&tx_solr* for *
- … 5 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /investments$ for *
- Added Disallow: /investments/ for *
- Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
- Removed Disallow: https://metlife-prod.adobecqms.net/* for *
- … 3 more change(s)
- Removed Disallow: /search/* for *
- Removed Disallow: /search?q=* for *