- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Added Disallow: /pd/*/af$ for *
- Added Disallow: /pd/*/af=$ for *
- Added Disallow: /pd/*/ar$ for *
- Added Disallow: /pd/*/az$ for *
- Added Disallow: /pd/*/be$ for *
- Added Disallow: /pd/*/bg$ for *
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Removed Disallow: /ReviewExpress- for *
- Removed Disallow: /ReviewExpress- for applebot
- Removed Disallow: /ReviewExpress- for bingbot
- 🤖 AI Removed Disallow: /ReviewExpress- for ChatGPT-User
- Removed Disallow: /ReviewExpress- for ChatGPT-User/2.0
- Removed Disallow: /ReviewExpress- for Gemini-Deep-Research
- … 3 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /my-trips/search?* for *
- Added Disallow: /mytrips/ for *
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap https://www.intuit.com/enterprise/blog/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /reengageoptout/* for *
- Removed sitemap https://www.kraken.com/sitemaps/buy-stocks.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘amazon-kendra’ entirely
- Blocked user-agent ‘amazon-QBusiness’ entirely
- Blocked user-agent ‘Apify-HTTP-Client’ entirely
- Blocked user-agent ‘bedrockbo’ entirely
- Blocked user-agent ‘bedrockbot’ entirely
- Blocked user-agent ‘BlueCart-API’ entirely
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /customers/how-shakers-uses-figma-to-align-teams-accelerate-delivery-and-elevate-craft/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/customers/how-shakers-uses-figma-to-align-teams-accelerate-delivery-and-elevate-craft/$ for CHATGPT-User
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /search.html for *