- Added sitemap https://www.thetradedesk.cn/sitemap.xml
- Removed sitemap https://www.thetradedesk.com/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2530snapshots
1357changes detected
272this week
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Removed Disallow: /nu/conta/caixinha-turbo for *
- Removed Disallow: /nu/investimentos/caixinhas for *
- Removed Disallow: /nu/conta/caixinha-turbo for DatoCmsSearchBot
- Removed Disallow: /nu/investimentos/caixinhas for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-bildbearbeitung/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/solutions/changer-couleur-ia/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/solutions/retouche-image-ia/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/solutions/vectoriser-image/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for Claude-SearchBot
- … 14 more change(s)
- Removed Disallow: /*/verifyEmail for *
- Removed Disallow: /verifyEmail for *
- Added Disallow: /research-survey for *
- Added Disallow: /rebuild for *
- Added Disallow: /studio for *
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /chat/* for *
- Removed Disallow: /search* for *
- Added sitemap https://www.pinterest.com/v3_sitemaps/board_link_sitemap_priority_segment_2026q3_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/priority_segment_2026q3_www.pinterest.com.xml
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /gtwy/accountservice/orchestration/accounts/* for *
- Added Disallow: /gtwy/accountservice/orchestration/signin/* for *
- Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/change/password for *
- Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/forgot/password for *
- Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/lookup for *
- Removed Disallow: /gtwy/api/accountservice/orchestration/accounts/update for *
- … 5 more change(s)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /*/news/press-releases/* for *
- Removed Disallow: /*/resource-center/research-report/* for *
- Removed Disallow: /*/resource-center/solution-brief/* for *
- Removed Disallow: /*/resources-industries/* for *
- Removed Disallow: /*/resources-products/* for *
- Removed Disallow: /*/resources-topics/* for *
- … 3 more change(s)
- Removed Disallow: /*?* for *
- Added Allow: /*/health$ for *
- Added Allow: /*/health/$ for *
- Added Allow: /health$ for *
- Added Allow: /health/$ for *
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)