- Added Disallow: /snb/ for *
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5439snapshots
4304changes detected
239this week
- Added Disallow: /video-center for *
- Added Disallow: /webinar-center for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- First snapshot of robots.txt archived
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
- Blocked user-agent ‘Comet’ entirely
- 🤖 AI Blocked Perplexity-User (Perplexity) entirely
- 🤖 AI Blocked PerplexityBot (Perplexity) entirely
- 🤖 AI Added Allow: /blog/2026-ai-report/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/agent-custom-tools-context-skills/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/code-on-the-figma-canvas/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/config-2026-recap/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/connecting-figma-and-weave/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/future-states/$ for CHATGPT-User
- … 45 more change(s)
- Added Disallow: /wireless/switch-and-save-b/ for *
- Added Disallow: /sites/default/files/white-papers/pfpt-jp-wp-securing-and-governing-AI-in-the-modern-enterprise-guide.pdf for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /lillydirect/oura for *
- Added Disallow: /lillydirect/oura-offer for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /app-ads.txt for *
- Added Disallow: /legal/vat-registration-numbers for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘Applebot’ entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘Baiduspider’ entirely
- Blocked user-agent ‘Baiduspider-image’ entirely
- Blocked user-agent ‘Baiduspider-video’ entirely
- … 13 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Added sitemap https://www.adobe.com/edu/sitemap-index.xml
- Removed sitemap https://www.adobe.com/edu-shared/assets/sitemap.xml
- Formatting/comment-only change (no semantic difference)