- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2625snapshots
1453changes detected
242this week
- 🤖 AI Removed Disallow: /start for Amazonbot
- 🤖 AI Removed Disallow: /start for Applebot-Extended
- 🤖 AI Removed Disallow: /start for CCBot
- 🤖 AI Removed Disallow: /start for ChatGPT-User
- 🤖 AI Removed Disallow: /start for Claude-SearchBot
- 🤖 AI Removed Disallow: /start for Claude-User
- … 12 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /uptime for *
- Added Allow: /*/import-to-chatgpt for *
- Added Allow: /*/remote for *
- Added Allow: /*/work for *
- Added Allow: /import-to-chatgpt for *
- Added Allow: /remote for *
- Added Allow: /work for *
- … 6 more change(s)
- Added Disallow: /epspayment/ for *
- Added Disallow: /payment/ for *
- Added Disallow: /tender/ for *
- Added Disallow: /tendercloud/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/customers/how-lollypop-takes-clients-from-brief-to-build-in-figma/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/the-figma-agent-is-here/$ for CHATGPT-User
- … 45 more change(s)
- Added Disallow: /kmu/ for *
- Added rules for user-agent ‘Amazonbot-Video’
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for Claude-SearchBot
- 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for OAI-SearchBot
- 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for PerplexityBot
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sitemap/ for *
- Removed sitemap https://www.snapchat.com/sitemaps/mit_tier1_legacy/sitemap-index.xml
- Removed sitemap https://www.snapchat.com/sitemaps/mit_tier1_mix/sitemap-index.xml
- Removed sitemap https://www.snapchat.com/sitemaps/mit_tier1_new/sitemap-index.xml
- Removed sitemap https://www.snapchat.com/sitemaps/mit_tier2_legacy/sitemap-index.xml
- Removed sitemap https://www.snapchat.com/sitemaps/mit_tier2_mix/sitemap-index.xml
- Removed sitemap https://www.snapchat.com/sitemaps/mit_tier2_new/sitemap-index.xml