Delta Air Lines / robots.txt
← back to delta.com · change detected 1d ago (2026-11-05T19:53:48Z) · modified +0 −58
What changed
- Removed all rules for user-agent ‘ai2-crawler’
- 🤖 AIRemoved rules for AI crawler Amazonbot (Amazon)
- 🤖 AIRemoved rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AIRemoved rules for AI crawler Applebot-Extended (Apple)
- 🤖 AIRemoved rules for AI crawler Bytespider (ByteDance)
- 🤖 AIRemoved rules for AI crawler CCBot (Common Crawl)
- 🤖 AIRemoved rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIRemoved rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIRemoved rules for AI crawler cohere-ai (Cohere)
- 🤖 AIRemoved rules for AI crawler Diffbot (Diffbot)
- 🤖 AIRemoved rules for AI crawler DuckAssistBot (DuckDuckGo)
- 🤖 AIRemoved rules for AI crawler FacebookBot (Meta)
- 🤖 AIRemoved rules for AI crawler Google-Extended (Google)
- 🤖 AIRemoved rules for AI crawler GPTBot (OpenAI)
- 🤖 AIRemoved rules for AI crawler meta-externalagent (Meta)
- 🤖 AIRemoved rules for AI crawler meta-externalfetcher (Meta)
- 🤖 AIRemoved rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIRemoved rules for AI crawler PerplexityBot (Perplexity)
- 🤖 AIRemoved rules for AI crawler YouBot (You.com)
Diff old (2026-11-03T19:10:24Z) → new (2026-11-05T19:53:48Z)
| @@ -1,61 +1,3 @@ | ||
| 1 | - # - LLM / AI Crawlers - Explicitly Permitted - | |
| 2 | - User-agent: GPTBot | |
| 3 | - Allow: / | |
| 4 | - | |
| 5 | - User-agent: ChatGPT-User | |
| 6 | - Allow: / | |
| 7 | - | |
| 8 | - User-agent: OAI-SearchBot | |
| 9 | - Allow: / | |
| 10 | - | |
| 11 | - User-agent: ClaudeBot | |
| 12 | - Allow: / | |
| 13 | - | |
| 14 | - User-agent: anthropic-ai | |
| 15 | - Allow: / | |
| 16 | - | |
| 17 | - User-agent: Google-Extended | |
| 18 | - Allow: / | |
| 19 | - | |
| 20 | - User-agent: FacebookBot | |
| 21 | - Allow: / | |
| 22 | - | |
| 23 | - User-agent: meta-externalagent | |
| 24 | - Allow: / | |
| 25 | - | |
| 26 | - User-agent: Meta-ExternalFetcher | |
| 27 | - Allow: / | |
| 28 | - | |
| 29 | - User-agent: Applebot-Extended | |
| 30 | - Allow: / | |
| 31 | - | |
| 32 | - User-agent: Amazonbot | |
| 33 | - Allow: / | |
| 34 | - | |
| 35 | - User-agent: PerplexityBot | |
| 36 | - Allow: / | |
| 37 | - | |
| 38 | - User-agent: CCBot | |
| 39 | - Allow: / | |
| 40 | - | |
| 41 | - User-agent: cohere-ai | |
| 42 | - Allow: / | |
| 43 | - | |
| 44 | - User-agent: Bytespider | |
| 45 | - Allow: / | |
| 46 | - | |
| 47 | - User-agent: DuckAssistBot | |
| 48 | - Allow: / | |
| 49 | - | |
| 50 | - User-agent: YouBot | |
| 51 | - Allow: / | |
| 52 | - | |
| 53 | - User-agent: Diffbot | |
| 54 | - Allow: / | |
| 55 | - | |
| 56 | - User-agent: ai2-crawler | |
| 57 | - Allow: / | |
| 58 | - | |
| 59 | 1 | # If you are reading this looking for a means of reporting a security issue, please see security.txt @ https://www.delta.com/us/en/legal/vulnerability-disclosure-guidelines |
| 60 | 2 | |
| 61 | 3 | User-agent: * |