Heineken / robots.txt
← back to heineken.com · change detected 9d ago (2026-10-28T17:03:01Z) · modified +10 −0
What changed
- 🤖 AIAdded rules for AI crawler CCBot (Common Crawl)
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
Diff old (2026-06-20T01:10:30Z) → new (2026-10-28T17:03:01Z)
| @@ -1,3 +1,13 @@ | ||
| 1 | + # AI/LLM crawlers | |
| 2 | + User-agent: OAI-SearchBot | |
| 3 | + User-agent: GPTBot | |
| 4 | + User-agent: ClaudeBot | |
| 5 | + User-agent: Claude-SearchBot | |
| 6 | + User-agent: Claude-User | |
| 7 | + User-agent: PerplexityBot | |
| 8 | + User-agent: CCBot | |
| 9 | + User-agent: Google-Extended | |
| 10 | + | |
| 1 | 11 | User-agent: * |
| 2 | 12 | Disallow: /umbraco/ |
| 3 | 13 |