- Added Disallow: /index%2Ephp/ for *
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /webinars/2026-ai-report/$ for CHATGPT-User
- 🤖 AI Removed Allow: /gallery/$ for CHATGPT-User
- 🤖 AI Added Allow: /webinars/2026-ai-report/$ for Claude-SearchBot
- 🤖 AI Removed Allow: /gallery/$ for Claude-SearchBot
- 🤖 AI Added Allow: /webinars/2026-ai-report/$ for Claude-User
- 🤖 AI Removed Allow: /gallery/$ for Claude-User
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /homefeed/ for AmazonAdBot
- Added Allow: /homefeed/* for AmazonAdBot
- Added Allow: /homefeed/ for Mediapartners-Google
- Added Allow: /homefeed/* for Mediapartners-Google
- Removed Allow: /*/atlas for *
- Removed Allow: /atlas for *
- Formatting/comment-only change (no semantic difference)
- Unblocked user-agent ‘*’
- Added Allow: / for *
- Added Disallow: /*/saved$ for *
- Added Disallow: /*/saved? for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://weather.com/api/location/sitemaps/en-US/sitemap.xml
- Added sitemap https://weather.com/api/payload/v1/sitemaps/en-US/sitemap.xml
- Blocked user-agent ‘*’ entirely
- Removed Allow: / for *
- Removed Disallow: /*/saved$ for *
- Removed Disallow: /*/saved? for *
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.metlife.com/investments/global/sitemap.xml
- robots.txt is no longer served (HTTP 404)
- Added Disallow: */directory/ for Bingbot
- Added Disallow: /api/recommend/embed_videos for Bingbot
- Added Disallow: /api/share/settings for Bingbot
- Added Disallow: /auth for Bingbot
- Added Disallow: /embed/@ for Bingbot
- Added Disallow: /embed/curated for Bingbot
- … 6 more change(s)
- Added Disallow: /*/saved$ for *
- Added Disallow: /*/saved? for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)