The Wall Street Journal / robots.txt
← back to wsj.com · change detected 2mo ago (2026-08-17T23:14:08Z) · modified +1 −0
What changed
- Added rules for user-agent ‘Wallie’
Diff old (2026-07-23T20:15:59Z) → new (2026-08-17T23:14:08Z)
| @@ -54,6 +54,7 @@ | ||
| 54 | 54 | User-agent: BomboraBot |
| 55 | 55 | User-agent: IllumaBot/1.0 |
| 56 | 56 | User-agent: Publication-Access-for-Facebook |
| 57 | + User-agent: Wallie | |
| 57 | 58 | |
| 58 | 59 | |
| 59 | 60 | User-agent: Screaming Frog SEO Spider |