- Added sitemap https://www.datadoghq.com/community/sitemap.xml
- Added sitemap https://www.datadoghq.com/newsletter/sitemap.xml
- Added sitemap https://www.datadoghq.com/page/sitemap.xml
- Added sitemap https://www.datadoghq.com/products/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
- 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
- Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261204_international_day_of_banks_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_chanukah_hanukkah_first_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_first_day_of_hanukkah_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_international_volunteer_day_for_economic_and_socia_www.pinterest.com.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘MistralAI-Training’ entirely
- Blocked user-agent ‘Webzio-Extended’ entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- 🤖 AI Removed Allow: /.well-known/amphtml/apikey.pub for ClaudeBot
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
- Added Disallow: Amazonbot for *
- Added Disallow: Amzn-SearchBot for *
- Added Disallow: Bytespider for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /de-de/blog/source-material-aitana-pico/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/source-material-aitana-pico/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/source-material-aitana-pico/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/blog/source-material-aitana-pico/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/blog/source-material-aitana-pico/$ for CHATGPT-User
- 🤖 AI Added Allow: /ko-kr/blog/source-material-aitana-pico/$ for CHATGPT-User
- … 29 more change(s)
- Removed sitemap https://suno.com/styles-sitemap.xml