- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2634snapshots
1462changes detected
251this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘BomboraBot’
- Added Disallow: /*?id=*&type=mdc_* for Citoid
- Added Disallow: /*?type=mdc_*&id=* for Citoid
- Added Disallow: /Date(* for Citoid
- Added Disallow: /acct/* for Citoid
- Added Disallow: /article/AP* for Citoid
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/ for *
- Added Disallow: /content/dam/vetrina/documenti/landing/anti-phishing/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for Claude-SearchBot
- 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for OAI-SearchBot
- 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for PerplexityBot
- Added rules for user-agent ‘*’
- Added sitemap https://www.lingyiwanwu.com/sitemap.xml
- Removed sitemap http://www.01.ai/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Added Allow: /$ for *
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /npas/ for *
- Added Disallow: /npasearch* for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /themes/ for *
- Formatting/comment-only change (no semantic difference)