GoDaddy / robots.txt
← back to godaddy.com · change detected 1d ago (2026-09-17T21:12:52Z) · modified +32 −11
What changed
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIAdded Allow: */domain-auctions/ for Claude-SearchBot
- 🤖 AIAdded Allow: */domainsearch/ for Claude-SearchBot
- 🤖 AIAdded Disallow: */domain-auctions/api/* for Claude-SearchBot
- 🤖 AIRemoved Allow: /domain-auctions/ for Claude-SearchBot
- 🤖 AIRemoved Allow: /domainsearch/ for Claude-SearchBot
- 🤖 AIAdded Allow: */domain-auctions/ for Claude-User
- 🤖 AIAdded Allow: */domainsearch/ for Claude-User
- 🤖 AIAdded Disallow: */domain-auctions/api/* for Claude-User
- 🤖 AIRemoved Allow: /domain-auctions/ for Claude-User
- 🤖 AIRemoved Allow: /domainsearch/ for Claude-User
- 🤖 AIAdded Allow: */domain-auctions/ for ClaudeBot
- 🤖 AIAdded Allow: */domainsearch/ for ClaudeBot
- 🤖 AIAdded Disallow: */domain-auctions/api/* for ClaudeBot
- 🤖 AIRemoved Allow: /domain-auctions/ for ClaudeBot
- 🤖 AIRemoved Allow: /domainsearch/ for ClaudeBot
- 🤖 AIAdded Allow: */domain-auctions/ for GPTBot
- 🤖 AIAdded Allow: */domainsearch/ for GPTBot
- 🤖 AIAdded Disallow: */domain-auctions/api/ for GPTBot
- 🤖 AIRemoved Allow: /domain-auctions/ for GPTBot
- 🤖 AIRemoved Allow: /domainsearch/ for GPTBot
- 🤖 AIAdded Allow: */domain-auctions/ for OAI-SearchBot
- 🤖 AIAdded Allow: */domainsearch/ for OAI-SearchBot
- 🤖 AIAdded Disallow: */domain-auctions/api/ for OAI-SearchBot
- 🤖 AIRemoved Allow: /domain-auctions/ for OAI-SearchBot
- 🤖 AIRemoved Allow: /domainsearch/ for OAI-SearchBot
- 🤖 AIAdded Allow: */domain-auctions/ for PerplexityBot
- 🤖 AIAdded Allow: */domainsearch/ for PerplexityBot
- 🤖 AIAdded Allow: /v1/search/ for PerplexityBot
- 🤖 AIAdded Disallow: */domain-auctions/api/* for PerplexityBot
- 🤖 AIRemoved Allow: /domain-auctions/ for PerplexityBot
Diff old (2026-09-16T20:58:51Z) → new (2026-09-17T21:12:52Z)
| @@ -2,33 +2,54 @@ | ||
| 2 | 2 | # robots.txt |
| 3 | 3 | # |
| 4 | 4 | |
| 5 | + User-agent: ChatGPT-User | |
| 6 | + Allow: */domain-auctions/ | |
| 7 | + Allow: */domainsearch/ | |
| 8 | + Allow: /v1/search/ | |
| 9 | + Disallow: */domain-auctions/api/* | |
| 10 | + | |
| 5 | 11 | User-agent: Claude-User |
| 6 | - Allow: /domain-auctions/ | |
| 7 | - Allow: /domainsearch/ | |
| 12 | + Allow: */domain-auctions/ | |
| 13 | + Allow: */domainsearch/ | |
| 8 | 14 | Allow: /v1/search/ |
| 15 | + Disallow: */domain-auctions/api/* | |
| 9 | 16 | |
| 10 | 17 | User-agent: Claude-SearchBot |
| 11 | - Allow: /domain-auctions/ | |
| 12 | - Allow: /domainsearch/ | |
| 18 | + Allow: */domain-auctions/ | |
| 19 | + Allow: */domainsearch/ | |
| 13 | 20 | Allow: /v1/search/ |
| 21 | + Disallow: */domain-auctions/api/* | |
| 14 | 22 | |
| 15 | 23 | User-agent: ClaudeBot |
| 16 | - Allow: /domain-auctions/ | |
| 17 | - Allow: /domainsearch/ | |
| 24 | + Allow: */domain-auctions/ | |
| 25 | + Allow: */domainsearch/ | |
| 18 | 26 | Allow: /v1/search/ |
| 27 | + Disallow: */domain-auctions/api/* | |
| 19 | 28 | |
| 20 | 29 | User-agent: PerplexityBot |
| 21 | - Allow: /domain-auctions/ | |
| 30 | + Allow: */domain-auctions/ | |
| 31 | + Allow: */domainsearch/ | |
| 32 | + Allow: /v1/search/ | |
| 33 | + Disallow: */domain-auctions/api/* | |
| 34 | + | |
| 35 | + User-agent: Perplexity-User | |
| 36 | + Allow: */domain-auctions/ | |
| 37 | + Allow: */domainsearch/ | |
| 38 | + Allow: /v1/search/ | |
| 39 | + Disallow: */domain-auctions/api/* | |
| 40 | + | |
| 22 | 41 | |
| 23 | 42 | User-agent: OAI-SearchBot |
| 24 | - Allow: /domain-auctions/ | |
| 25 | - Allow: /domainsearch/ | |
| 43 | + Allow: */domain-auctions/ | |
| 44 | + Allow: */domainsearch/ | |
| 26 | 45 | Allow: /v1/search/ |
| 46 | + Disallow: */domain-auctions/api/ | |
| 27 | 47 | |
| 28 | 48 | User-agent: GPTBot |
| 29 | - Allow: /domain-auctions/ | |
| 30 | - Allow: /domainsearch/ | |
| 49 | + Allow: */domain-auctions/ | |
| 50 | + Allow: */domainsearch/ | |
| 31 | 51 | Allow: /v1/search/ |
| 52 | + Disallow: */domain-auctions/api/ | |
| 32 | 53 | |
| 33 | 54 | |
| 34 | 55 | User-agent: * |