Sophos / robots.txt
← back to sophos.com · change detected 17d ago (2026-10-21T02:33:48Z) · modified +7 −1
What changed
- Added rules for user-agent ‘Amzn-SearchBot’
- Added rules for user-agent ‘Amzn-User’
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- 🤖 AIRemoved Allow: / for Amazonbot
Diff old (2026-06-20T14:27:25Z) → new (2026-10-21T02:33:48Z)
| @@ -72,8 +72,14 @@ | ||
| 72 | 72 | User-agent: xAI-Bot |
| 73 | 73 | Allow: / |
| 74 | 74 | |
| 75 | + User-agent: Amzn-SearchBot | |
| 76 | + Allow: / | |
| 77 | + | |
| 78 | + User-agent: Amzn-User | |
| 79 | + Allow: / | |
| 80 | + | |
| 75 | 81 | User-agent: Amazonbot |
| 76 | - Allow: / | |
| 82 | + Disallow: / | |
| 77 | 83 | |
| 78 | 84 | User-agent: FirecrawlAgent |
| 79 | 85 | Allow: / |