- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /Paginas/test1.aspx for *
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Claude-User (Anthropic) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
- 🤖 AI Added Allow: /de/product/ for anthropic-ai
- 🤖 AI Added Allow: /de/video-uploader for anthropic-ai
- 🤖 AI Added Allow: /es/product/ for anthropic-ai
- 🤖 AI Added Allow: /es/video-uploader for anthropic-ai
- 🤖 AI Added Allow: /fr/product/ for anthropic-ai
- 🤖 AI Added Allow: /fr/video-uploader for anthropic-ai
- … 45 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /es-es/soluciones/agente-de-diseno-ia/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/soluciones/cambiador-de-estilo-ia/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/soluciones/editor-de-fotos-por-lotes/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/soluciones/generador-de-flujo-de-pago/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/soluciones/indicacion-para-editar/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/soluciones/revision-de-diseno/$ for CHATGPT-User
- … 45 more change(s)
- 🤖 AI Added Allow: /de/upload/ for anthropic-ai
- 🤖 AI Added Allow: /es/upload/ for anthropic-ai
- 🤖 AI Added Allow: /fr/upload/ for anthropic-ai
- 🤖 AI Added Allow: /jp/upload/ for anthropic-ai
- 🤖 AI Added Allow: /ko/upload/ for anthropic-ai
- 🤖 AI Added Allow: /pt-br/upload/ for anthropic-ai
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for ChatGPT Operator
- 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for ChatGPT-User
- 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for Google-Extended
- 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for GPTBot
- 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for OAI-SearchBot
- 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for PerplexityBot
- Added Allow: /es/blog for *
- Added Allow: /ja/blog for *
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /games/*?*clid= for *
- Added Disallow: /games/*?*deviceid= for *
- Added Disallow: /games/*?*flags= for *
- Added Disallow: /games/*?*inapp_id= for *
- Added Disallow: /games/*?*k50id= for *
- Added Disallow: /games/*?*lang= for *
- … 33 more change(s)