Snapchat / robots.txt
← back to snapchat.com · change detected 23h ago (2026-07-17T18:17:47Z) · modified +42 −23
What changed
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- Removed all rules for user-agent ‘Perplexity‑User’
- Removed Disallow: /entertainment/ for *
- 🤖 AIUnblocked ChatGPT-User (OpenAI)
- 🤖 AIAdded Allow: / for ChatGPT-User
- 🤖 AIAdded Disallow: /explore/ for ChatGPT-User
- 🤖 AIAdded Disallow: /getreplies for ChatGPT-User
- 🤖 AIUnblocked DuckAssistBot (DuckDuckGo)
- 🤖 AIAdded Allow: / for DuckAssistBot
- 🤖 AIAdded Disallow: /explore/ for DuckAssistBot
- 🤖 AIAdded Disallow: /getreplies for DuckAssistBot
- Removed Disallow: /entertainment/ for Googlebot
- Removed Disallow: /entertainment/ for Meta-WebIndexer
- 🤖 AIUnblocked OAI-SearchBot (OpenAI)
- 🤖 AIAdded Allow: / for OAI-SearchBot
- 🤖 AIAdded Disallow: /explore/ for OAI-SearchBot
- 🤖 AIAdded Disallow: /getreplies for OAI-SearchBot
- 🤖 AIUnblocked PerplexityBot (Perplexity)
- 🤖 AIAdded Allow: / for PerplexityBot
- 🤖 AIAdded Disallow: /explore/ for PerplexityBot
- 🤖 AIAdded Disallow: /getreplies for PerplexityBot
Diff old (2026-07-15T17:22:35Z) → new (2026-07-17T18:17:47Z)
| @@ -4,12 +4,10 @@ | ||
| 4 | 4 | Disallow: /static_files/deletion_request.pdf |
| 5 | 5 | Disallow: /static/gla/ |
| 6 | 6 | Disallow: /invite/ |
| 7 | - Disallow: /entertainment/ | |
| 8 | 7 | |
| 9 | 8 | # Meta-WebIndexer |
| 10 | 9 | User-agent: Meta-WebIndexer |
| 11 | 10 | Allow: / |
| 12 | - Disallow: /entertainment/ | |
| 13 | 11 | |
| 14 | 12 | # Meta-ExternalAgent |
| 15 | 13 | User-agent: Meta-ExternalAgent |
| @@ -18,6 +16,48 @@ | ||
| 18 | 16 | # Meta-ExternalFetcher |
| 19 | 17 | User-agent: Meta-ExternalFetcher |
| 20 | 18 | Disallow: / |
| 19 | + | |
| 20 | + # OAI-SearchBot | |
| 21 | + User-agent: OAI-SearchBot | |
| 22 | + Allow: / | |
| 23 | + Disallow: /getreplies | |
| 24 | + Disallow: /explore/ | |
| 25 | + | |
| 26 | + # ChatGPT-User | |
| 27 | + User-agent: ChatGPT-User | |
| 28 | + Allow: / | |
| 29 | + Disallow: /getreplies | |
| 30 | + Disallow: /explore/ | |
| 31 | + | |
| 32 | + # PerplexityBot | |
| 33 | + User-agent: PerplexityBot | |
| 34 | + Allow: / | |
| 35 | + Disallow: /getreplies | |
| 36 | + Disallow: /explore/ | |
| 37 | + | |
| 38 | + # Perplexity-User | |
| 39 | + User-agent: Perplexity-User | |
| 40 | + Allow: / | |
| 41 | + Disallow: /getreplies | |
| 42 | + Disallow: /explore/ | |
| 43 | + | |
| 44 | + # Claude-SearchBot | |
| 45 | + User-agent: Claude-SearchBot | |
| 46 | + Allow: / | |
| 47 | + Disallow: /getreplies | |
| 48 | + Disallow: /explore/ | |
| 49 | + | |
| 50 | + # Claude-User | |
| 51 | + User-agent: Claude-User | |
| 52 | + Allow: / | |
| 53 | + Disallow: /getreplies | |
| 54 | + Disallow: /explore/ | |
| 55 | + | |
| 56 | + # DuckAssistBot | |
| 57 | + User-agent: DuckAssistBot | |
| 58 | + Allow: / | |
| 59 | + Disallow: /getreplies | |
| 60 | + Disallow: /explore/ | |
| 21 | 61 | |
| 22 | 62 | # AI2Bot |
| 23 | 63 | User-agent: AI2Bot |
| @@ -51,10 +91,6 @@ | ||
| 51 | 91 | User-agent: CCBot |
| 52 | 92 | Disallow: / |
| 53 | 93 | |
| 54 | - # ChatGPT-User | |
| 55 | - User-agent: ChatGPT-User | |
| 56 | - Disallow: / | |
| 57 | - | |
| 58 | 94 | # Claude-Web |
| 59 | 95 | User-agent: Claude-Web |
| 60 | 96 | Disallow: / |
| @@ -79,10 +115,6 @@ | ||
| 79 | 115 | User-agent: Diffbot |
| 80 | 116 | Disallow: / |
| 81 | 117 | |
| 82 | - # DuckAssistBot | |
| 83 | - User-agent: DuckAssistBot | |
| 84 | - Disallow: / | |
| 85 | - | |
| 86 | 118 | # FacebookBot |
| 87 | 119 | User-agent: FacebookBot |
| 88 | 120 | Disallow: / |
| @@ -95,7 +127,6 @@ | ||
| 95 | 127 | User-agent: Googlebot |
| 96 | 128 | Disallow: /getreplies |
| 97 | 129 | Disallow: /explore/ |
| 98 | - Disallow: /entertainment/ | |
| 99 | 130 | |
| 100 | 131 | # Google-Extended |
| 101 | 132 | User-agent: Google-Extended |
| @@ -141,10 +172,6 @@ | ||
| 141 | 172 | User-agent: Kangaroo Bot |
| 142 | 173 | Disallow: / |
| 143 | 174 | |
| 144 | - # OAI-SearchBot | |
| 145 | - User-agent: OAI-SearchBot | |
| 146 | - Disallow: / | |
| 147 | - | |
| 148 | 175 | # omgili |
| 149 | 176 | User-agent: omgili |
| 150 | 177 | Disallow: / |
| @@ -155,14 +182,6 @@ | ||
| 155 | 182 | |
| 156 | 183 | # PanguBot |
| 157 | 184 | User-agent: PanguBot |
| 158 | - Disallow: / | |
| 159 | - | |
| 160 | - # PerplexityBot | |
| 161 | - User-agent: PerplexityBot | |
| 162 | - Disallow: / | |
| 163 | - | |
| 164 | - # Perplexity‑User | |
| 165 | - User-agent: Perplexity‑User | |
| 166 | 185 | Disallow: / |
| 167 | 186 | |
| 168 | 187 | # PetalBot |