USA Today / robots.txt
← back to usatoday.com · change detected 2mo ago (2026-08-11T21:11:35Z) · modified +6 −0
What changed
- Added Allow: /story/shopping/ for ChatGPT Operator
- 🤖 AIAdded Allow: /story/shopping/ for ChatGPT-User
- 🤖 AIAdded Allow: /story/shopping/ for Google-Extended
- 🤖 AIAdded Allow: /story/shopping/ for GPTBot
- 🤖 AIAdded Allow: /story/shopping/ for OAI-SearchBot
- 🤖 AIAdded Allow: /story/shopping/ for PerplexityBot
Diff old (2026-08-07T19:48:33Z) → new (2026-08-11T21:11:35Z)
| @@ -36,6 +36,7 @@ | ||
| 36 | 36 | Allow: /story/special/contributor-content/ |
| 37 | 37 | Allow: /money/money-management/ |
| 38 | 38 | Allow: /shopping/ |
| 39 | + Allow: /story/shopping/ | |
| 39 | 40 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 40 | 41 | |
| 41 | 42 | User-agent: Googlebot-News |
| @@ -53,6 +54,7 @@ | ||
| 53 | 54 | Allow: /story/special/contributor-content/ |
| 54 | 55 | Allow: /money/money-management/ |
| 55 | 56 | Allow: /shopping/ |
| 57 | + Allow: /story/shopping/ | |
| 56 | 58 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 57 | 59 | |
| 58 | 60 | User-agent: CCBot |
| @@ -64,6 +66,7 @@ | ||
| 64 | 66 | Allow: /story/special/contributor-content/ |
| 65 | 67 | Allow: /money/money-management/ |
| 66 | 68 | Allow: /shopping/ |
| 69 | + Allow: /story/shopping/ | |
| 67 | 70 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 68 | 71 | |
| 69 | 72 | User-agent: DataForSeoBot |
| @@ -108,6 +111,7 @@ | ||
| 108 | 111 | Allow: /story/special/contributor-content/ |
| 109 | 112 | Allow: /money/money-management/ |
| 110 | 113 | Allow: /shopping/ |
| 114 | + Allow: /story/shopping/ | |
| 111 | 115 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 112 | 116 | |
| 113 | 117 | User-agent: Scrapy |
| @@ -131,6 +135,7 @@ | ||
| 131 | 135 | Allow: /story/special/contributor-content/ |
| 132 | 136 | Allow: /money/money-management/ |
| 133 | 137 | Allow: /shopping/ |
| 138 | + Allow: /story/shopping/ | |
| 134 | 139 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 135 | 140 | |
| 136 | 141 | User-agent: Bytespider |
| @@ -364,6 +369,7 @@ | ||
| 364 | 369 | Allow: /story/special/contributor-content/ |
| 365 | 370 | Allow: /money/money-management/ |
| 366 | 371 | Allow: /shopping/ |
| 372 | + Allow: /story/shopping/ | |
| 367 | 373 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 368 | 374 | |
| 369 | 375 | User-agent: ChatGPT-PageFetcher |