GitHub / robots.txt
← back to github.com · change detected 2h ago (2026-11-07T08:19:09Z) · modified +18 −1
What changed
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIRemoved rules for AI crawler anthropic-ai (Anthropic)
- Added Allow: /enterprise/contact/data-residency?* for *
- Added Allow: /enterprise/contact?* for *
- Added Allow: /features/codespaces?* for *
- Added Allow: /features/copilot/plans?* for *
- Added Allow: /security/advanced-security/demo?* for *
- Added Allow: /security/contact-sales?* for *
- Added Allow: /security/plans?* for *
- 🤖 AIAdded Allow: /enterprise/contact/data-residency?* for ClaudeBot
- 🤖 AIAdded Allow: /enterprise/contact?* for ClaudeBot
- 🤖 AIAdded Allow: /features/codespaces?* for ClaudeBot
- 🤖 AIAdded Allow: /features/copilot/plans?* for ClaudeBot
- 🤖 AIAdded Allow: /security/advanced-security/demo?* for ClaudeBot
- 🤖 AIAdded Allow: /security/contact-sales?* for ClaudeBot
- 🤖 AIAdded Allow: /security/plans?* for ClaudeBot
- 🤖 AIAdded Allow: /enterprise/contact/data-residency?* for GPTBot
- 🤖 AIAdded Allow: /enterprise/contact?* for GPTBot
- 🤖 AIAdded Allow: /features/codespaces?* for GPTBot
- 🤖 AIAdded Allow: /features/copilot/plans?* for GPTBot
- 🤖 AIAdded Allow: /security/advanced-security/demo?* for GPTBot
- 🤖 AIAdded Allow: /security/contact-sales?* for GPTBot
- 🤖 AIAdded Allow: /security/plans?* for GPTBot
- 🤖 AIAdded Allow: /enterprise/contact/data-residency?* for OAI-SearchBot
- 🤖 AIAdded Allow: /enterprise/contact?* for OAI-SearchBot
- 🤖 AIAdded Allow: /features/codespaces?* for OAI-SearchBot
- 🤖 AIAdded Allow: /features/copilot/plans?* for OAI-SearchBot
- 🤖 AIAdded Allow: /security/advanced-security/demo?* for OAI-SearchBot
- 🤖 AIAdded Allow: /security/contact-sales?* for OAI-SearchBot
- 🤖 AIAdded Allow: /security/plans?* for OAI-SearchBot
- 🤖 AIAdded Allow: /enterprise/contact/data-residency?* for PerplexityBot
- 🤖 AIAdded Allow: /enterprise/contact?* for PerplexityBot
- 🤖 AIAdded Allow: /features/codespaces?* for PerplexityBot
- 🤖 AIAdded Allow: /features/copilot/plans?* for PerplexityBot
- 🤖 AIAdded Allow: /security/advanced-security/demo?* for PerplexityBot
- 🤖 AIAdded Allow: /security/contact-sales?* for PerplexityBot
- 🤖 AIAdded Allow: /security/plans?* for PerplexityBot
Diff old (2026-10-20T08:19:06Z) → new (2026-11-07T08:19:09Z)
| @@ -4,13 +4,23 @@ | ||
| 4 | 4 | |
| 5 | 5 | User-agent: GPTBot |
| 6 | 6 | User-agent: OAI-SearchBot |
| 7 | + User-agent: ChatGPT-User | |
| 7 | 8 | User-agent: ClaudeBot |
| 8 | - User-agent: anthropic-ai | |
| 9 | + User-agent: Claude-SearchBot | |
| 10 | + User-agent: Claude-User | |
| 9 | 11 | User-agent: PerplexityBot |
| 12 | + User-agent: Perplexity-User | |
| 10 | 13 | Crawl-delay: 1 |
| 11 | 14 | Allow: /*?tab=achievements&achievement=* |
| 12 | 15 | Allow: /marketplace/*?tab=readme$ |
| 13 | 16 | Allow: /marketplace/*?tab=transparency$ |
| 17 | + Allow: /enterprise/contact?* | |
| 18 | + Allow: /enterprise/contact/data-residency?* | |
| 19 | + Allow: /features/codespaces?* | |
| 20 | + Allow: /features/copilot/plans?* | |
| 21 | + Allow: /security/advanced-security/demo?* | |
| 22 | + Allow: /security/contact-sales?* | |
| 23 | + Allow: /security/plans?* | |
| 14 | 24 | Disallow: /account-login |
| 15 | 25 | Disallow: /copilot/ |
| 16 | 26 | Disallow: /copilot/c/ |
| @@ -361,6 +371,13 @@ | ||
| 361 | 371 | Allow: /*?tab=achievements&achievement=* |
| 362 | 372 | Allow: /marketplace/*?tab=readme$ |
| 363 | 373 | Allow: /marketplace/*?tab=transparency$ |
| 374 | + Allow: /enterprise/contact?* | |
| 375 | + Allow: /enterprise/contact/data-residency?* | |
| 376 | + Allow: /features/codespaces?* | |
| 377 | + Allow: /features/copilot/plans?* | |
| 378 | + Allow: /security/advanced-security/demo?* | |
| 379 | + Allow: /security/contact-sales?* | |
| 380 | + Allow: /security/plans?* | |
| 364 | 381 | Disallow: /account-login |
| 365 | 382 | Disallow: /copilot/ |
| 366 | 383 | Disallow: /copilot/c/ |