Best Buy / robots.txt
← back to bestbuy.com · change detected 2d ago (2026-11-05T07:45:14Z) · modified +14 −4
What changed
- 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘Brightbot’
- 🤖 AIAdded rules for AI crawler Diffbot (Diffbot)
- 🤖 AIAdded rules for AI crawler DuckAssistBot (DuckDuckGo)
- Added rules for user-agent ‘Gemini-AI’
- Added rules for user-agent ‘GoogleAgent-Mariner’
- Added rules for user-agent ‘GrokBot’
- 🤖 AIAdded rules for AI crawler meta-externalagent (Meta)
- 🤖 AIAdded rules for AI crawler meta-externalfetcher (Meta)
- Added rules for user-agent ‘webzio’
- Added rules for user-agent ‘xAI-Grok’
- Removed all rules for user-agent ‘Gemeni-AI’
- 🤖 AIAdded Allow: / for AmazonBot
- 🤖 AIRemoved Disallow: /[1.1] for Anthropic-AI
- Added Allow: / for Applebot
- Blocked user-agent ‘BigSurAI’ entirely
- Removed Disallow: / [2.1] for BigSurAI
- 🤖 AIRemoved Disallow: /[1.1] for Bytespider
- 🤖 AIRemoved Disallow: /[1.1] for CCBot
- 🤖 AIAdded Allow: / for ChatGPT-User
- 🤖 AIAdded Allow: / for Claude-User
- 🤖 AIRemoved Disallow: /[1.1] for Claude-Web
- 🤖 AIAdded Allow: / for Claudebot
- Blocked user-agent ‘DotBot’ entirely
- Removed Disallow: / [2.1] for DotBot
- Added Allow: / for Gemini-Deep-Research
- Blocked user-agent ‘GetIntentCrawler’ entirely
- Removed Disallow: / [2.1] for GetIntentCrawler
- 🤖 AIAdded Allow: / for Google-Extended
- 🤖 AIAdded Allow: / for GPTBot
- Removed Disallow: /[1.1] for GPTBot-Web
- Blocked user-agent ‘MJ12bot’ entirely
- Removed Disallow: / [2.1] for MJ12bot
- Blocked user-agent ‘MyCentralAIScraper’ entirely
- Removed Disallow: / [2.1] for MyCentralAIScraper
- 🤖 AIAdded Allow: / for OAI-SearchBot
- 🤖 AIAdded Allow: / for perplexity-user
- 🤖 AIAdded Allow: / for PerplexityBot
- Blocked user-agent ‘TerraCotta’ entirely
- Removed Disallow: / [2.1] for TerraCotta
- Blocked user-agent ‘thesis-research-bot’ entirely
- Removed Disallow: / [2.1] for thesis-research-bot
Diff old (2026-09-17T15:11:14Z) → new (2026-11-05T07:45:14Z)
| @@ -10,17 +10,28 @@ | ||
| 10 | 10 | # ================================================== |
| 11 | 11 | User-agent: OAI-SearchBot |
| 12 | 12 | User-agent: ChatGPT-User |
| 13 | - User-agent: Perplexity-User | |
| 13 | + User-agent: perplexity-user | |
| 14 | 14 | User-agent: GPTBot |
| 15 | 15 | User-agent: Claude-User |
| 16 | 16 | User-agent: Claudebot |
| 17 | 17 | User-agent: PerplexityBot |
| 18 | 18 | User-agent: Applebot |
| 19 | + User-agent: Applebot-Extended | |
| 19 | 20 | User-agent: Google-Extended |
| 20 | - User-agent: Gemeni-AI | |
| 21 | + User-agent: Gemini-AI | |
| 21 | 22 | User-agent: Gemini-Deep-Research |
| 23 | + User-agent: GoogleAgent-Mariner | |
| 22 | 24 | User-agent: AmazonBot |
| 25 | + User-agent: GrokBot | |
| 26 | + User-agent: xAI-Grok | |
| 27 | + User-agent: Diffbot | |
| 28 | + User-agent: Meta-ExternalAgent | |
| 29 | + User-agent: Meta-ExternalFetcher | |
| 30 | + User-agent: DuckAssistBot | |
| 31 | + User-agent: Brightbot | |
| 32 | + User-agent: webzio | |
| 23 | 33 | |
| 34 | + Allow: / | |
| 24 | 35 | Allow: /site/ |
| 25 | 36 | Allow: /discover-learn/ |
| 26 | 37 | |
| @@ -182,7 +193,6 @@ | ||
| 182 | 193 | User-agent: Anthropic-AI |
| 183 | 194 | User-agent: Claude-Web |
| 184 | 195 | User-agent: Bytespider |
| 185 | - Disallow: /[1.1] | |
| 186 | 196 | |
| 187 | 197 | # ================================================== |
| 188 | 198 | # BUCKET 3: UNIFIED WEB ENGINES & UNIVERSAL DEFAULTS |
| @@ -417,7 +427,7 @@ | ||
| 417 | 427 | User-agent: TerraCotta |
| 418 | 428 | User-agent: BigSurAI |
| 419 | 429 | User-agent: MyCentralAIScraper |
| 420 | - Disallow: / [2.1] | |
| 430 | + Disallow: / | |
| 421 | 431 | |
| 422 | 432 | # ================================================== |
| 423 | 433 | # DATA SITEMAPS |