- 🤖 AI Added Allow: /horlogerie for ChatGPT-User
- 🤖 AI Added Allow: /joaillerie for ChatGPT-User
- 🤖 AI Added Allow: /mode-homme for ChatGPT-User
- 🤖 AI Added Allow: /horlogerie for OAI-SearchBot
- 🤖 AI Added Allow: /joaillerie for OAI-SearchBot
- 🤖 AI Added Allow: /mode-homme for OAI-SearchBot
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Added Allow: /students/2026 for *
- 🤖 AI Added rules for AI crawler DuckAssistBot (DuckDuckGo)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Removed Disallow: /web.config for *
- Added sitemap https://www.westlake.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /content-transition for *
- Removed Disallow: /empresas-blocos for *
- Removed Disallow: /uv-blocos for *
- Removed Disallow: /content-transition for DatoCmsSearchBot
- Removed Disallow: /empresas-blocos for DatoCmsSearchBot
- Removed Disallow: /uv-blocos for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Added rules for user-agent ‘AdmiralBot’
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
- 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
- Added rules for user-agent ‘Applebot’
- Added rules for user-agent ‘BigSurAI’
- Added rules for user-agent ‘Bingbot’
- 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
- … 26 more change(s)
- Added Allow: /documents/d/nossa-energia/ for *
- Added Allow: /o/adaptive-media/image/ for *
- Added Disallow: /*/c/portal/ for *
- Added Disallow: /*?category=* for *
- Added Disallow: /*?delta=* for *
- Added Disallow: /*?p_p_* for *
- … 43 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /corporate-governance/supplier-info for *
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /resource-library/best-ai-app-for-logo-design/$ for CHATGPT-User
- 🤖 AI Added Allow: /solutions/ai-breadcrumb-generator/$ for CHATGPT-User
- 🤖 AI Added Allow: /solutions/ai-design-changelog-generator/$ for CHATGPT-User
- 🤖 AI Added Allow: /solutions/ai-skeleton-loader-generator/$ for CHATGPT-User
- 🤖 AI Added Allow: /solutions/ai-split-test-variant-generator/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/best-ai-app-for-logo-design/$ for Claude-SearchBot
- … 19 more change(s)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘Brightbot’ entirely
- Blocked user-agent ‘Diffbot-User’ entirely
- Blocked user-agent ‘ExaBot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)