NeuralCrawl

USA Today / robots.txt

← back to usatoday.com · change detected 13d ago (2026-07-22T19:49:29Z) · modified +6 −0

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-07-22T19:49:29Z)

@@ -34,6 +34,7 @@
3434  Disallow: /
3535  Allow: /contributor-content/
3636  Allow: /story/special/contributor-content/
37+ Allow: /money/money-management/
3738  Allow: /shopping/
3839 
3940  User-agent: Googlebot-News
@@ -49,6 +50,7 @@
4950  Disallow: /
5051  Allow: /contributor-content/
5152  Allow: /story/special/contributor-content/
53+ Allow: /money/money-management/
5254  Allow: /shopping/
5355 
5456  User-agent: CCBot
@@ -58,6 +60,7 @@
5860  Disallow: /
5961  Allow: /contributor-content/
6062  Allow: /story/special/contributor-content/
63+ Allow: /money/money-management/
6164  Allow: /shopping/
6265 
6366  User-agent: DataForSeoBot
@@ -100,6 +103,7 @@
100103  Disallow: /
101104  Allow: /contributor-content/
102105  Allow: /story/special/contributor-content/
106+ Allow: /money/money-management/
103107  Allow: /shopping/
104108 
105109  User-agent: Scrapy
@@ -121,6 +125,7 @@
121125  Disallow: /
122126  Allow: /contributor-content/
123127  Allow: /story/special/contributor-content/
128+ Allow: /money/money-management/
124129  Allow: /shopping/
125130 
126131  User-agent: Bytespider
@@ -352,6 +357,7 @@
352357  Disallow: /
353358  Allow: /contributor-content/
354359  Allow: /story/special/contributor-content/
360+ Allow: /money/money-management/
355361  Allow: /shopping/
356362 
357363  User-agent: ChatGPT-PageFetcher