- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.snapchat.com/profiles-sitemap.xml
- Added sitemap https://www.snapchat.com/sitemaps/profile/sitemap-index.xml
- Removed sitemap https://www.snapchat.com/add/sitemap.xml
- Removed sitemap https://www.snapchat.com/sitemaps/add/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘SiteimproveBot’
- Added Disallow: (empty) for *
- Removed Allow: /*_catalogs/*.css for *
- Removed Allow: /*_catalogs/*.js for *
- Removed Allow: /*_layouts/*.css for *
- Removed Allow: /*_layouts/*.eot for *
- … 24 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /index%2ephp/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /resource-library/ai-design-prompts/$ for CHATGPT-User
- 🤖 AI Removed Allow: /de-de/gallery/$ for CHATGPT-User
- 🤖 AI Removed Allow: /es-es/gallery/$ for CHATGPT-User
- 🤖 AI Removed Allow: /es-la/gallery/$ for CHATGPT-User
- 🤖 AI Removed Allow: /fr-fr/gallery/$ for CHATGPT-User
- 🤖 AI Removed Allow: /it-it/gallery/$ for CHATGPT-User
- … 45 more change(s)
- 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
- Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for *
- Removed Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for *
- Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for DotBot
- Removed Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for DotBot
- Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for sistrix
- … 3 more change(s)
- Added sitemap https://apnews.com/author-sitemap-content.xml
- Added Allow: /*/maps$ for *
- Added Allow: /maps$ for *
- Added sitemap https://group.ntt/sitemap.xml
- Removed sitemap https://www.group.ntt/sitemap.xml
- 🤖 AI Added Allow: /*.ghtm$ for Google-Extended
- 🤖 AI Removed Allow: /.ghtm$ for Google-Extended
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Added Disallow: /index%2Ephp/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /webinars/2026-ai-report/$ for CHATGPT-User
- 🤖 AI Removed Allow: /gallery/$ for CHATGPT-User
- 🤖 AI Added Allow: /webinars/2026-ai-report/$ for Claude-SearchBot
- 🤖 AI Removed Allow: /gallery/$ for Claude-SearchBot
- 🤖 AI Added Allow: /webinars/2026-ai-report/$ for Claude-User
- 🤖 AI Removed Allow: /gallery/$ for Claude-User
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /homefeed/ for AmazonAdBot
- Added Allow: /homefeed/* for AmazonAdBot
- Added Allow: /homefeed/ for Mediapartners-Google
- Added Allow: /homefeed/* for Mediapartners-Google