The Wall Street Journal / robots.txt
← back to wsj.com · change detected 1mo ago (2026-07-06T20:20:23Z) · modified +5 −6
What changed
- Added rules for user-agent ‘BomboraBot’
- Added Disallow: /*?id=*&type=mdc_* for Citoid
- Added Disallow: /*?type=mdc_*&id=* for Citoid
- Added Disallow: /Date(* for Citoid
- Added Disallow: /acct/* for Citoid
- Added Disallow: /article/AP* for Citoid
- Added Disallow: /article/BT-CO* for Citoid
- Added Disallow: /article/DN-CO* for Citoid
- Added Disallow: /article/HUG* for Citoid
- Added Disallow: /article/PR-CO* for Citoid
- Added Disallow: /article_email/* for Citoid
- Added Disallow: /articles/BT-CO* for Citoid
- Added Disallow: /articles/DN-CO* for Citoid
- Added Disallow: /articles/PR-CO* for Citoid
- Added Disallow: /articles/SB2* for Citoid
- Added Disallow: /articles/SB3* for Citoid
- Added Disallow: /articles/SB4* for Citoid
- Added Disallow: /articles/the-meaning-behind-juneteenth-11592413234 for Citoid
- Added Disallow: /auth/sso/proxy-login* for Citoid
- Added Disallow: /buyside/search-results?*term=* for Citoid
- Added Disallow: /catchup/* for Citoid
- Added Disallow: /client/ for Citoid
- Added Disallow: /emailservice/* for Citoid
- Added Disallow: /emailsignup/* for Citoid
- Added Disallow: /insetsrv/v1/* for Citoid
- Added Disallow: /login/* for Citoid
- Added Disallow: /market-data/quotes/*/options/* for Citoid
- Added Disallow: /marketing/* for Citoid
- Added Disallow: /msgcenter/* for Citoid
- Added Disallow: /news/articles/BT-CO* for Citoid
- Added Disallow: /news/articles/DN-CO* for Citoid
- Added Disallow: /news/articles/PR-CO* for Citoid
- Added Disallow: /news/articles/SB2* for Citoid
- Added Disallow: /news/articles/SB3* for Citoid
- Added Disallow: /news/articles/SB4* for Citoid
- Added Disallow: /pdf/documents/* for Citoid
- Added Disallow: /public/article/* for Citoid
- Added Disallow: /public/page/0_0_WP_RT_MARKETING.html for Citoid
- Added Disallow: /public/page/news-media-marketing.html for Citoid
- Added Disallow: /public/page/wsj-x-marketing.html for Citoid
- Added Disallow: /public/resources/documents/* for Citoid
- Added Disallow: /public/search* for Citoid
- Added Disallow: /public/search/ for Citoid
- Added Disallow: /search* for Citoid
- Added Disallow: /setup/* for Citoid
- Added Disallow: /subscribe/?inttrackingCode=* for Citoid
- Added Disallow: /subscribe/?template=* for Citoid
- Added Disallow: /user/* for Citoid
- Added Disallow: /user/fpd/api/* for Citoid
- Added Disallow: /video/search/* for Citoid
- ... and 539 more change(s)
Diff old (2026-06-20T01:10:30Z) → new (2026-07-06T20:20:23Z)
| @@ -23,7 +23,6 @@ | ||
| 23 | 23 | User-agent: AdsBot-Google-Mobile-Apps |
| 24 | 24 | User-agent: Storebot-Google |
| 25 | 25 | User-agent: google-read-aloud |
| 26 | - User-agent: mediapartners-google | |
| 27 | 26 | User-agent: bingbot |
| 28 | 27 | User-agent: msnbot |
| 29 | 28 | User-agent: bingpreview |
| @@ -61,6 +60,7 @@ | ||
| 61 | 60 | User-agent: SISTRIX |
| 62 | 61 | User-agent: botify |
| 63 | 62 | User-agent: Chrome-Lighthouse |
| 63 | + User-agent: BomboraBot | |
| 64 | 64 | |
| 65 | 65 | |
| 66 | 66 | User-agent: ChatGPT-User |
| @@ -83,11 +83,6 @@ | ||
| 83 | 83 | |
| 84 | 84 | |
| 85 | 85 | Allow: / |
| 86 | - | |
| 87 | - User-agent: mediapartners-google | |
| 88 | - Disallow: / | |
| 89 | - Allow: /watchlist | |
| 90 | - | |
| 91 | 86 | Disallow: /article_email/* |
| 92 | 87 | Disallow: /user/* |
| 93 | 88 | Disallow: /pdf/documents/* |
| @@ -141,6 +136,10 @@ | ||
| 141 | 136 | Disallow: /subscribe/?inttrackingCode=* |
| 142 | 137 | Disallow: /subscribe/?template=* |
| 143 | 138 | |
| 139 | + User-agent: mediapartners-google | |
| 140 | + Disallow: / | |
| 141 | + Allow: /watchlist | |
| 142 | + | |
| 144 | 143 | Sitemap: https://www.wsj.com/sitemap.xml |
| 145 | 144 | Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml |
| 146 | 145 | Sitemap: https://www.wsj.com/wsj_video_recent.xml |