- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- 🤖 AI Added Allow: /de-de/loesungen/checkout-flow-generator/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/design-review/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-design-agent/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-stilwechsler/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/prompt-zum-bearbeiten/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/stapelbildeditor/$ for CHATGPT-User
- … 45 more change(s)
- Added Allow: /llms.txt for *
- 🤖 AI Unblocked ClaudeBot (Anthropic)
- 🤖 AI Added Allow: / for ClaudeBot
- 🤖 AI Unblocked GPTBot (OpenAI)
- 🤖 AI Added Allow: / for GPTBot
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.penskeautomotive.com/sitemap-cars.xml
- Added sitemap https://www.penskeautomotive.com/sitemap.xml
- Removed sitemap https://www.penskeautomotive.com/sitemap.xmlSitemap: https://www.penskeautomotive.com/sitemap-cars.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.lilly.com/ch/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /pro$ for Google-Extended
- 🤖 AI Added Allow: /pro/ for Google-Extended
- 🤖 AI Added Allow: /de-de/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /ko-kr/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
- … 22 more change(s)
- Added Disallow: /*/silent.html for 360Spider
- Added Disallow: /silent.html for 360Spider
- 🤖 AI Added Disallow: /*/silent.html for Amazonbot
- 🤖 AI Added Disallow: /silent.html for Amazonbot
- Added Disallow: /*/silent.html for archive.org_bot
- Added Disallow: /silent.html for archive.org_bot
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Added Disallow: /abbvie-welcome.html for *
- Added Disallow: /join-us/welcome-to-abbvie.html for *
- Removed Disallow: /content/abbvie-com2/us/en/abbvie-welcome.html for *
- Removed Disallow: /content/abbvie-com2/us/en/join-us/welcome-to-abbvie.html for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /search?q=%23 for Bingbot
- Removed Disallow: /search?q= for Bingbot
- Added Allow: /search?q=%23 for Googlebot
- Removed Disallow: /search?q= for Googlebot
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added Disallow: /user/ for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)