- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /realestateandhomes-detail/*/gallery/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Removed Disallow: /*/?*Ntt= for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /de-de/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
- … 45 more change(s)
- Added Disallow: /alice/chat/ for *
- Added Disallow: /alice/chat/ for yandex
- Blocked user-agent ‘AgentTimes’ entirely
- Blocked user-agent ‘AI2Bot-DeepResearchEval’ entirely
- Blocked user-agent ‘AIWebIndex’ entirely
- Blocked user-agent ‘amazon-kendra’ entirely
- Blocked user-agent ‘amazon-QBusiness’ entirely
- Blocked user-agent ‘AmazonBuyForMe’ entirely
- … 45 more change(s)
- Added Disallow: /api/gated-resource/ for *
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://tidal.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- 🤖 AI Added Disallow: /fetch for Amazonbot
- 🤖 AI Added Disallow: /proxy for Amazonbot
- 🤖 AI Added Disallow: /sharedlibs/ for Amazonbot
- 🤖 AI Added Disallow: /fetch for ChatGPT-User
- 🤖 AI Added Disallow: /proxy for ChatGPT-User
- 🤖 AI Added Disallow: /sharedlibs/ for ChatGPT-User
- … 12 more change(s)