- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Removed Disallow: /hs/preferences-center/ for *
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /Search for *
- Added Disallow: /es/locations for *
- Added Disallow: /pl/locations for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘baiduspider’
- Removed all rules for user-agent ‘bingbot’
- 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
- Removed all rules for user-agent ‘Googlebot’
- Removed all rules for user-agent ‘MSNbot’
- 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
- … 19 more change(s)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Removed Disallow: /Search for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /hs/preferences-center/ for *
- Added rules for user-agent ‘baiduspider’
- Added rules for user-agent ‘bingbot’
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- Added rules for user-agent ‘Googlebot’
- Added rules for user-agent ‘MSNbot’
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- … 18 more change(s)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)