Fortra / robots.txt
← back to fortra.com · change detected 1mo ago (2026-07-01T00:16:40Z) · modified +45 −0
What changed
- 🤖 AIAdded rules for AI crawler Amazonbot (Amazon)
- 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
- 🤖 AIAdded rules for AI crawler Bytespider (ByteDance)
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- 🤖 AIBlocked Google-Extended (Google) entirely
- Added rules for user-agent ‘GoogleOther’
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- Added rules for user-agent ‘TikTokSpider’
- Added Allow: / for *
Diff old (2026-06-20T14:27:25Z) → new (2026-07-01T00:16:40Z)
| @@ -59,6 +59,51 @@ | ||
| 59 | 59 | Disallow: /index.php/media/oembed |
| 60 | 60 | Disallow: /index.php/*/media/oembed |
| 61 | 61 | |
| 62 | + # === AI Crawler === | |
| 63 | + | |
| 64 | + # OpenAI | |
| 65 | + User-agent: GPTBot | |
| 66 | + Disallow: / | |
| 67 | + | |
| 68 | + User-agent: ChatGPT-User | |
| 69 | + Allow: / | |
| 70 | + | |
| 71 | + # Anthropic | |
| 72 | + User-agent: ClaudeBot | |
| 73 | + Disallow: / | |
| 74 | + | |
| 75 | + # Google AI training | |
| 76 | + User-agent: Google-Extended | |
| 77 | + Disallow: / | |
| 78 | + | |
| 79 | + # Google general AI crawling | |
| 80 | + User-agent: GoogleOther | |
| 81 | + Allow: / | |
| 82 | + | |
| 83 | + # Perplexity | |
| 84 | + User-agent: PerplexityBot | |
| 85 | + Allow: / | |
| 86 | + | |
| 87 | + # Amazon (no clear training split → allow) | |
| 88 | + User-agent: Amazonbot | |
| 89 | + Allow: / | |
| 90 | + | |
| 91 | + # Apple (no clear training split → allow) | |
| 92 | + User-agent: Applebot-Extended | |
| 93 | + Allow: / | |
| 94 | + | |
| 95 | + # ByteDance / TikTok (no clear training split → allow) | |
| 96 | + User-agent: Bytespider | |
| 97 | + Allow: / | |
| 98 | + | |
| 99 | + User-agent: TikTokSpider | |
| 100 | + Allow: / | |
| 101 | + | |
| 102 | + # Fallback | |
| 103 | + User-agent: * | |
| 104 | + Allow: / | |
| 105 | + | |
| 106 | + | |
| 62 | 107 | # |
| 63 | 108 | # Upstream Robots Additions |
| 64 | 109 | # |