USA Today / robots.txt
← back to usatoday.com · change detected 14d ago (2026-09-05T11:12:24Z) · modified +12 −6
What changed
- Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
- Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
- Removed Allow: /branded-story/ for ChatGPT Operator
- 🤖 AIAdded Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
- 🤖 AIAdded Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
- 🤖 AIRemoved Allow: /branded-story/ for ChatGPT-User
- 🤖 AIAdded Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for Google-Extended
- 🤖 AIAdded Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for Google-Extended
- 🤖 AIRemoved Allow: /branded-story/ for Google-Extended
- 🤖 AIAdded Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for GPTBot
- 🤖 AIAdded Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for GPTBot
- 🤖 AIRemoved Allow: /branded-story/ for GPTBot
- 🤖 AIAdded Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for OAI-SearchBot
- 🤖 AIAdded Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for OAI-SearchBot
- 🤖 AIRemoved Allow: /branded-story/ for OAI-SearchBot
- 🤖 AIAdded Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for PerplexityBot
- 🤖 AIAdded Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for PerplexityBot
- 🤖 AIRemoved Allow: /branded-story/ for PerplexityBot
Diff old (2026-09-04T16:59:43Z) → new (2026-09-05T11:12:24Z)
| @@ -35,10 +35,11 @@ | ||
| 35 | 35 | Allow: /contributor-content/ |
| 36 | 36 | Allow: /story/special/contributor-content/ |
| 37 | 37 | Allow: /money/money-management/ |
| 38 | - Allow: /branded-story/ | |
| 39 | 38 | Allow: /shopping/ |
| 40 | 39 | Allow: /story/shopping/ |
| 41 | 40 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 41 | + Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ | |
| 42 | + Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ | |
| 42 | 43 | |
| 43 | 44 | User-agent: Googlebot-News |
| 44 | 45 | Disallow: /story/sponsor-story/ |
| @@ -54,10 +55,11 @@ | ||
| 54 | 55 | Allow: /contributor-content/ |
| 55 | 56 | Allow: /story/special/contributor-content/ |
| 56 | 57 | Allow: /money/money-management/ |
| 57 | - Allow: /branded-story/ | |
| 58 | 58 | Allow: /shopping/ |
| 59 | 59 | Allow: /story/shopping/ |
| 60 | 60 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 61 | + Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ | |
| 62 | + Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ | |
| 61 | 63 | |
| 62 | 64 | User-agent: CCBot |
| 63 | 65 | Disallow: / |
| @@ -67,10 +69,11 @@ | ||
| 67 | 69 | Allow: /contributor-content/ |
| 68 | 70 | Allow: /story/special/contributor-content/ |
| 69 | 71 | Allow: /money/money-management/ |
| 70 | - Allow: /branded-story/ | |
| 71 | 72 | Allow: /shopping/ |
| 72 | 73 | Allow: /story/shopping/ |
| 73 | 74 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 75 | + Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ | |
| 76 | + Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ | |
| 74 | 77 | |
| 75 | 78 | User-agent: DataForSeoBot |
| 76 | 79 | Disallow: / |
| @@ -113,10 +116,11 @@ | ||
| 113 | 116 | Allow: /contributor-content/ |
| 114 | 117 | Allow: /story/special/contributor-content/ |
| 115 | 118 | Allow: /money/money-management/ |
| 116 | - Allow: /branded-story/ | |
| 117 | 119 | Allow: /shopping/ |
| 118 | 120 | Allow: /story/shopping/ |
| 119 | 121 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 122 | + Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ | |
| 123 | + Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ | |
| 120 | 124 | |
| 121 | 125 | User-agent: Scrapy |
| 122 | 126 | Disallow: / |
| @@ -138,10 +142,11 @@ | ||
| 138 | 142 | Allow: /contributor-content/ |
| 139 | 143 | Allow: /story/special/contributor-content/ |
| 140 | 144 | Allow: /money/money-management/ |
| 141 | - Allow: /branded-story/ | |
| 142 | 145 | Allow: /shopping/ |
| 143 | 146 | Allow: /story/shopping/ |
| 144 | 147 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 148 | + Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ | |
| 149 | + Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ | |
| 145 | 150 | |
| 146 | 151 | User-agent: Bytespider |
| 147 | 152 | Disallow: / |
| @@ -373,10 +378,11 @@ | ||
| 373 | 378 | Allow: /contributor-content/ |
| 374 | 379 | Allow: /story/special/contributor-content/ |
| 375 | 380 | Allow: /money/money-management/ |
| 376 | - Allow: /branded-story/ | |
| 377 | 381 | Allow: /shopping/ |
| 378 | 382 | Allow: /story/shopping/ |
| 379 | 383 | Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ |
| 384 | + Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ | |
| 385 | + Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ | |
| 380 | 386 | |
| 381 | 387 | User-agent: ChatGPT-PageFetcher |
| 382 | 388 | Disallow: / |