- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2614snapshots
1442changes detected
231this week
- Removed sitemap https://www.intuit.com/enterprise/blog/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /reengageoptout/* for *
- Removed sitemap https://www.kraken.com/sitemaps/buy-stocks.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘amazon-kendra’ entirely
- Blocked user-agent ‘amazon-QBusiness’ entirely
- Blocked user-agent ‘Apify-HTTP-Client’ entirely
- Blocked user-agent ‘bedrockbo’ entirely
- Blocked user-agent ‘bedrockbot’ entirely
- Blocked user-agent ‘BlueCart-API’ entirely
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /customers/how-shakers-uses-figma-to-align-teams-accelerate-delivery-and-elevate-craft/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/customers/how-shakers-uses-figma-to-align-teams-accelerate-delivery-and-elevate-craft/$ for CHATGPT-User
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /search.html for *
- Added Disallow: /sports/world-cup/fifa-world-cup-2026-bracket-challenge/group for *
- Removed Disallow: /sports/world-cup/fifa-world-cup-2026-bracket-challenge/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /markets/energy/crude-oil/commodity-direct-spot-crude-markets.html for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /boas-vindas-a-cconta-nu-empresas for *
- Added Disallow: /boas-vindas-a-conta-nu-empresas for *
- Added Disallow: /boas-vindas-a-cconta-nu-empresas for DatoCmsSearchBot
- Added Disallow: /boas-vindas-a-conta-nu-empresas for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*&fbclid= for *
- Added Disallow: /*&gb= for *
- Added Disallow: /*&playlist= for *
- Added Disallow: /*&q= for *
- Added Disallow: /*&traffic_source= for *
- Added Disallow: /*?fbclid= for *
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)