- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Removed Allow: /$ for ClaudeBot
- 🤖 AI Removed Allow: /africa$ for ClaudeBot
- 🤖 AI Removed Allow: /africa/$ for ClaudeBot
- 🤖 AI Removed Allow: /asia$ for ClaudeBot
- 🤖 AI Removed Allow: /asia/$ for ClaudeBot
- 🤖 AI Removed Allow: /canada$ for ClaudeBot
- … 32 more change(s)
- Added Disallow: /_/semrush_regular_1_blank/ for *
- Removed Disallow: /_/empty_blank/ for *
- Removed Disallow: /_/site_info_blank/ for *
- Removed Disallow: /_/toolkit_blank/ for *
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘ahrefsbot’ entirely
- Blocked user-agent ‘cazoodlebot’ entirely
- Blocked user-agent ‘dotbot/1.0’ entirely
- Blocked user-agent ‘gigabot’ entirely
- Blocked user-agent ‘mj12bot’ entirely
- Blocked user-agent ‘sogou inst spider’ entirely
- … 20 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sokresultat/?query= for *
- Removed Disallow: /search/?query= for *
- Added sitemap https://www.regeringen.se/sitemap.xml
- Removed sitemap https://www.government.se/en/sitemap.xml
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Added Disallow: /content/dam/tcom/personal/consumer-advice/pdf/branded-calling-product-terms.pdf for *
- Removed all rules for user-agent ‘ahrefsbot’
- Removed all rules for user-agent ‘cazoodlebot’
- Removed all rules for user-agent ‘dotbot/1.0’
- Removed all rules for user-agent ‘gigabot’
- Removed all rules for user-agent ‘mj12bot’
- Removed all rules for user-agent ‘sogou inst spider’
- … 20 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /search/?query= for *
- Removed Disallow: /sokresultat/?query= for *
- Added sitemap https://www.government.se/en/sitemap.xml
- Removed sitemap https://www.regeringen.se/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /shellenergy/othersolutions.html for *
- Added Disallow: /shellenergy/othersolutions/ for *
- Added Disallow: /shellenergy/shell-energy-europe/natural-gas-portfolio.html for *
- Added Disallow: /shellenergy/shell-energy-europe/natural-gas-portfolio/ for *
- Added Disallow: /what-we-do/renewable-power.html for *
- Added Disallow: /what-we-do/renewable-power/ for *
- … 14 more change(s)
- Removed sitemap https://www.datadoghq.com/community/sitemap.xml
- Removed sitemap https://www.datadoghq.com/newsletter/sitemap.xml
- Removed sitemap https://www.datadoghq.com/page/sitemap.xml
- Removed sitemap https://www.datadoghq.com/products/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)