The Wall Street Journal / robots.txt
← back to wsj.com · change detected 2mo ago (2026-09-08T18:06:18Z) · modified +1 −3
What changed
- Unblocked user-agent ‘mediapartners-google’
- Removed Allow: /watchlist for mediapartners-google
Diff old (2026-08-17T23:14:08Z) → new (2026-09-08T18:06:18Z)
| @@ -55,6 +55,7 @@ | ||
| 55 | 55 | User-agent: IllumaBot/1.0 |
| 56 | 56 | User-agent: Publication-Access-for-Facebook |
| 57 | 57 | User-agent: Wallie |
| 58 | + User-agent: mediapartners-google | |
| 58 | 59 | |
| 59 | 60 | |
| 60 | 61 | User-agent: Screaming Frog SEO Spider |
| @@ -140,9 +141,6 @@ | ||
| 140 | 141 | Disallow: /subscribe/?inttrackingCode=* |
| 141 | 142 | Disallow: /subscribe/?template=* |
| 142 | 143 | |
| 143 | - User-agent: mediapartners-google | |
| 144 | - Disallow: / | |
| 145 | - Allow: /watchlist | |
| 146 | 144 | |
| 147 | 145 | Sitemap: https://www.wsj.com/sitemap.xml |
| 148 | 146 | Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml |