NeuralCrawl

USA Today / robots.txt

← back to usatoday.com · change detected 1mo ago (2026-09-29T00:37:36Z) · modified +27 −6

What changed

Diff old (2026-09-05T17:13:05Z) → new (2026-09-29T00:37:36Z)

@@ -32,13 +32,16 @@
3232 
3333  User-agent: ChatGPT-User
3434  Disallow: /
35+ Allow: /branded-story/
3536  Allow: /contributor-content/
37+ Allow: /story/branded-story/
3638  Allow: /story/special/contributor-content/
3739  Allow: /money/money-management/
38- Allow: /branded-story/
3940  Allow: /shopping/
4041  Allow: /story/shopping/
4142  Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/
43+ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/
44+ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/
4245 
4346  User-agent: Googlebot-News
4447  Disallow: /story/sponsor-story/
@@ -51,26 +54,32 @@
5154 
5255  User-agent: GPTBot
5356  Disallow: /
57+ Allow: /branded-story/
5458  Allow: /contributor-content/
59+ Allow: /story/branded-story/
5560  Allow: /story/special/contributor-content/
5661  Allow: /money/money-management/
57- Allow: /branded-story/
5862  Allow: /shopping/
5963  Allow: /story/shopping/
6064  Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/
65+ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/
66+ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/
6167 
6268  User-agent: CCBot
6369  Disallow: /
6470 
6571  User-agent: Google-Extended
6672  Disallow: /
73+ Allow: /branded-story/
6774  Allow: /contributor-content/
75+ Allow: /story/branded-story/
6876  Allow: /story/special/contributor-content/
6977  Allow: /money/money-management/
70- Allow: /branded-story/
7178  Allow: /shopping/
7279  Allow: /story/shopping/
7380  Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/
81+ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/
82+ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/
7483 
7584  User-agent: DataForSeoBot
7685  Disallow: /
@@ -110,13 +119,16 @@
110119 
111120  User-agent: PerplexityBot
112121  Disallow: /
122+ Allow: /branded-story/
113123  Allow: /contributor-content/
124+ Allow: /story/branded-story/
114125  Allow: /story/special/contributor-content/
115126  Allow: /money/money-management/
116- Allow: /branded-story/
117127  Allow: /shopping/
118128  Allow: /story/shopping/
119129  Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/
130+ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/
131+ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/
120132 
121133  User-agent: Scrapy
122134  Disallow: /
@@ -135,13 +147,16 @@
135147 
136148  User-agent: OAI-SearchBot
137149  Disallow: /
150+ Allow: /branded-story/
138151  Allow: /contributor-content/
152+ Allow: /story/branded-story/
139153  Allow: /story/special/contributor-content/
140154  Allow: /money/money-management/
141- Allow: /branded-story/
142155  Allow: /shopping/
143156  Allow: /story/shopping/
144157  Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/
158+ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/
159+ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/
145160 
146161  User-agent: Bytespider
147162  Disallow: /
@@ -370,13 +385,16 @@
370385 
371386  User-agent: ChatGPT Operator
372387  Disallow: /
388+ Allow: /branded-story/
373389  Allow: /contributor-content/
390+ Allow: /story/branded-story/
374391  Allow: /story/special/contributor-content/
375392  Allow: /money/money-management/
376- Allow: /branded-story/
377393  Allow: /shopping/
378394  Allow: /story/shopping/
379395  Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/
396+ Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/
397+ Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/
380398 
381399  User-agent: ChatGPT-PageFetcher
382400  Disallow: /
@@ -955,6 +973,9 @@
955973  User-agent: AmazonAdBot
956974  Allow: /
957975 
976+ User-agent: NewsBank.com
977+ Allow: /obituaries/
978+
958979  Sitemap: https://www.usatoday.com/news-sitemap.xml
959980  Sitemap: https://www.usatoday.com/web-sitemap-index.xml
960981  Sitemap: https://www.usatoday.com/video-sitemap-index.xml