- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Removed sitemap https://www.molinahealthcare.com/sitemap.xml
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*?*app_name= for *
- Added Disallow: /*?*matchcenter= for *
- Removed Disallow: /*?app_name= for *
- Removed Disallow: /*?matchcenter= for *
- 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for Claude-User
- 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for OAI-SearchBot
- 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for PerplexityBot
- Added sitemap https://lumalabs.ai/models/sitemap.xml
- Added sitemap https://www.cohesity.com/sitemap.xml
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: *UCPthyssenkruppAG* for *
- Removed Disallow: /_api for *
- Removed Disallow: /_media/ for *
- Removed Disallow: /assets.files/ for *
- Removed sitemap https://www.thyssenkrupp.com/sitemap.xml
- Added Disallow: /upload/*.m3u8 for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Added sitemap https://itau.com.br/investimentos/analises/sitemap.xml
- Added sitemap https://itau.com.br/investimentos/blog/sitemap.xml
- Added sitemap https://itau.com.br/investimentos/carteiras-recomendadas/sitemap.xml
- Added sitemap https://itau.com.br/investimentos/impactos-de-mercado/sitemap.xml
- Added sitemap https://itau.com.br/investimentos/noticias/sitemap.xml
- Added sitemap https://itau.com.br/investimentos/servicos/sitemap.xml
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /_tcm/ for *
- Removed Disallow: /akamai/ for *
- Removed Disallow: /byt/ for *
- Removed Disallow: /byt4/pub/ for *
- Removed Disallow: /capable-suvs/ for *
- Removed Disallow: /configurator/*zipcode= for *
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)