The Wall Street Journal / robots.txt
← back to wsj.com · change detected 17h ago (2026-11-06T20:13:58Z) · modified +1 −2
What changed
- Added rules for user-agent ‘NewsRoom.BI’
Diff old (2026-09-08T18:06:18Z) → new (2026-11-06T20:13:58Z)
| @@ -56,7 +56,7 @@ | ||
| 56 | 56 | User-agent: Publication-Access-for-Facebook |
| 57 | 57 | User-agent: Wallie |
| 58 | 58 | User-agent: mediapartners-google |
| 59 | - | |
| 59 | + User-agent: NewsRoom.BI | |
| 60 | 60 | |
| 61 | 61 | User-agent: Screaming Frog SEO Spider |
| 62 | 62 | User-agent: AhrefsBot |
| @@ -84,7 +84,6 @@ | ||
| 84 | 84 | User-agent: EpvzCrawl6194680250 |
| 85 | 85 | User-agent: Citoid |
| 86 | 86 | User-agent: ZoteroTranslationServer |
| 87 | - | |
| 88 | 87 | |
| 89 | 88 | |
| 90 | 89 | Allow: / |