Der Spiegel / robots.txt
← back to spiegel.de · change detected 1mo ago (2026-09-29T12:52:06Z) · modified +36 −6
What changed
- 🤖 AIAdded Allow: /ausland/ for Claude-SearchBot
- 🤖 AIAdded Allow: /familie/ for Claude-SearchBot
- 🤖 AIAdded Allow: /gesundheit/ for Claude-SearchBot
- 🤖 AIAdded Allow: /partnerschaft/ for Claude-SearchBot
- 🤖 AIAdded Allow: /psychologie/ for Claude-SearchBot
- 🤖 AIAdded Allow: /reise/ for Claude-SearchBot
- 🤖 AIAdded Allow: /stil/ for Claude-SearchBot
- 🤖 AIAdded Allow: /tests/ for Claude-SearchBot
- 🤖 AIAdded Allow: /ausland/ for Claude-User
- 🤖 AIAdded Allow: /familie/ for Claude-User
- 🤖 AIAdded Allow: /gesundheit/ for Claude-User
- 🤖 AIAdded Allow: /partnerschaft/ for Claude-User
- 🤖 AIAdded Allow: /psychologie/ for Claude-User
- 🤖 AIAdded Allow: /reise/ for Claude-User
- 🤖 AIAdded Allow: /stil/ for Claude-User
- 🤖 AIAdded Allow: /tests/ for Claude-User
- 🤖 AIAdded Allow: /ausland/ for Claude-Web
- 🤖 AIAdded Allow: /familie/ for Claude-Web
- 🤖 AIAdded Allow: /gesundheit/ for Claude-Web
- 🤖 AIAdded Allow: /partnerschaft/ for Claude-Web
- 🤖 AIAdded Allow: /psychologie/ for Claude-Web
- 🤖 AIAdded Allow: /reise/ for Claude-Web
- 🤖 AIAdded Allow: /stil/ for Claude-Web
- 🤖 AIAdded Allow: /tests/ for Claude-Web
Diff old (2026-09-08T12:01:45Z) → new (2026-09-29T12:52:06Z)
| @@ -35,6 +35,42 @@ | ||
| 35 | 35 | Allow: /tests/ |
| 36 | 36 | Disallow: / |
| 37 | 37 | |
| 38 | + # TLP-7042: Testweise Freischaltung Claude-Web fuer ausgewaehlte Bereiche | |
| 39 | + User-agent: Claude-Web | |
| 40 | + Allow: /ausland/ | |
| 41 | + Allow: /partnerschaft/ | |
| 42 | + Allow: /gesundheit/ | |
| 43 | + Allow: /familie/ | |
| 44 | + Allow: /reise/ | |
| 45 | + Allow: /psychologie/ | |
| 46 | + Allow: /stil/ | |
| 47 | + Allow: /tests/ | |
| 48 | + Disallow: / | |
| 49 | + | |
| 50 | + # TLP-7042: Testweise Freischaltung Claude-SearchBot fuer ausgewaehlte Bereiche | |
| 51 | + User-agent: Claude-SearchBot | |
| 52 | + Allow: /ausland/ | |
| 53 | + Allow: /partnerschaft/ | |
| 54 | + Allow: /gesundheit/ | |
| 55 | + Allow: /familie/ | |
| 56 | + Allow: /reise/ | |
| 57 | + Allow: /psychologie/ | |
| 58 | + Allow: /stil/ | |
| 59 | + Allow: /tests/ | |
| 60 | + Disallow: / | |
| 61 | + | |
| 62 | + # TLP-7042: Testweise Freischaltung Claude-User fuer ausgewaehlte Bereiche | |
| 63 | + User-agent: Claude-User | |
| 64 | + Allow: /ausland/ | |
| 65 | + Allow: /partnerschaft/ | |
| 66 | + Allow: /gesundheit/ | |
| 67 | + Allow: /familie/ | |
| 68 | + Allow: /reise/ | |
| 69 | + Allow: /psychologie/ | |
| 70 | + Allow: /stil/ | |
| 71 | + Allow: /tests/ | |
| 72 | + Disallow: / | |
| 73 | + | |
| 38 | 74 | # ========================================== |
| 39 | 75 | # KI-CRAWLER & KI-TRAINING (SPERRE) |
| 40 | 76 | # ========================================== |
| @@ -42,13 +78,7 @@ | ||
| 42 | 78 | Disallow: / |
| 43 | 79 | User-agent: anthropic-ai |
| 44 | 80 | Disallow: / |
| 45 | - User-agent: Claude-Web | |
| 46 | - Disallow: / | |
| 47 | 81 | User-agent: ClaudeBot |
| 48 | - Disallow: / | |
| 49 | - User-agent: Claude-SearchBot | |
| 50 | - Disallow: / | |
| 51 | - User-agent: Claude-User | |
| 52 | 82 | Disallow: / |
| 53 | 83 | User-agent: CloudVertexBot |
| 54 | 84 | Disallow: / |