- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2571snapshots
1398changes detected
225this week
- 🤖 AI Added Allow: /blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/ai-creative-process/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/ai-tools-for-graphic-design/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/claude-skill-design-system/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/design-operations/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/user-centered-design-questions/$ for CHATGPT-User
- … 22 more change(s)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler Google-Extended (Google)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Amazonbot (Amazon)
- 🤖 AI Added Allow: /ajax/bootloader-endpoint/ for Amazonbot
- 🤖 AI Added Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Amazonbot
- 🤖 AI Added Allow: /careers/ for Amazonbot
- 🤖 AI Added Allow: /safetycheck/ for Amazonbot
- 🤖 AI Added Disallow: /*/plugins/* for Amazonbot
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘Google-InspectionTool’
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml