Check Point / robots.txt
← back to checkpoint.com · change detected 1mo ago (2026-10-01T01:22:45Z) · modified +31 −0
What changed
- Added rules for user-agent ‘AndiBot’
- 🤖 AIAdded rules for AI crawler CCBot (Common Crawl)
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler Claude-Web (Anthropic)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- Added rules for user-agent ‘ExaBot’
- Added rules for user-agent ‘FirecrawlAgent’
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- Added rules for user-agent ‘PhindBot’
Diff old (2026-09-18T21:31:03Z) → new (2026-10-01T01:22:45Z)
| @@ -51,4 +51,35 @@ | ||
| 51 | 51 | Disallow: *tribe_geofence=* |
| 52 | 52 | Disallow: *tribe_organizer=* |
| 53 | 53 | |
| 54 | + # AI Agents & LLM Crawlers | |
| 55 | + User-agent: GPTBot | |
| 56 | + Allow: / | |
| 57 | + User-agent: OAI-SearchBot | |
| 58 | + Allow: / | |
| 59 | + User-agent: ChatGPT-User | |
| 60 | + Allow: / | |
| 61 | + User-agent: ClaudeBot | |
| 62 | + Allow: / | |
| 63 | + User-agent: Claude-Web | |
| 64 | + Allow: / | |
| 65 | + User-agent: PerplexityBot | |
| 66 | + Allow: / | |
| 67 | + User-agent: Perplexity-User | |
| 68 | + Allow: / | |
| 69 | + User-agent: Google-Extended | |
| 70 | + Allow: / | |
| 71 | + User-agent: CCBot | |
| 72 | + Allow: / | |
| 73 | + User-agent: FirecrawlAgent | |
| 74 | + Allow: / | |
| 75 | + User-agent: ExaBot | |
| 76 | + Allow: / | |
| 77 | + User-agent: PhindBot | |
| 78 | + Allow: / | |
| 79 | + User-agent: AndiBot | |
| 80 | + Allow: / | |
| 81 | + | |
| 54 | 82 | Sitemap: https://www.checkpoint.com/sitemap_index.xml |
| 83 | + | |
| 84 | + # AI Discovery Endpoints (Documentation Reference) | |
| 85 | + LLMS: https://www.checkpoint.com/llms.txt |