- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /*/design-school/ for anthropic-ai
- 🤖 AI Added Allow: /design-school/ for anthropic-ai
- 🤖 AI Added Allow: /*/design-school/ for Applebot-Extended
- 🤖 AI Added Allow: /design-school/ for Applebot-Extended
- 🤖 AI Added Allow: /*/design-school/ for CCBot
- 🤖 AI Added Allow: /design-school/ for CCBot
- … 6 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /content/dam/im/assets/publication/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for AmazonBot
- Removed Disallow: */site/searchpage.jsp?st=* for Applebot
- 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for ChatGPT-User
- 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for Claude-User
- 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for Claudebot
- Removed Disallow: */site/searchpage.jsp?st=* for Gemeni-AI
- … 6 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.algolia.com/doc/sitemap.xml
- Added sitemap https://www.algolia.com/sitemap.xml
- 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for Claude-SearchBot
- 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for Claude-User
- 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for OAI-SearchBot
- 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for PerplexityBot
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /*.ics$ for *
- Removed Disallow: /ajax/ for *
- Removed Disallow: /de/glossar-details/ for *
- Removed Disallow: /dynamic/fragment/ for *
- Removed Disallow: /en/glossary-details/ for *
- Removed Disallow: /ui-com-16-de/ for *
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sg/wealth-insights/ for *
- Added Disallow: /hs/preferences-center/ for *