The Wall Street Journal / robots.txt
← back to wsj.com · change detected 13d ago (2026-07-23T20:15:59Z) · modified +6 −1
What changed
- Added rules for user-agent ‘IllumaBot/1.0’
- Added rules for user-agent ‘Publication-Access-for-Facebook’
Diff old (2026-07-06T20:20:23Z) → new (2026-07-23T20:15:59Z)
| @@ -51,6 +51,9 @@ | ||
| 51 | 51 | User-agent: pubmatic |
| 52 | 52 | User-agent: smartologybot |
| 53 | 53 | User-agent: parselybot |
| 54 | + User-agent: BomboraBot | |
| 55 | + User-agent: IllumaBot/1.0 | |
| 56 | + User-agent: Publication-Access-for-Facebook | |
| 54 | 57 | |
| 55 | 58 | |
| 56 | 59 | User-agent: Screaming Frog SEO Spider |
| @@ -60,7 +63,7 @@ | ||
| 60 | 63 | User-agent: SISTRIX |
| 61 | 64 | User-agent: botify |
| 62 | 65 | User-agent: Chrome-Lighthouse |
| 63 | - User-agent: BomboraBot | |
| 66 | + | |
| 64 | 67 | |
| 65 | 68 | |
| 66 | 69 | User-agent: ChatGPT-User |
| @@ -152,3 +155,5 @@ | ||
| 152 | 155 | Sitemap: https://www.wsj.com/wsj_quote_index_sitemap.xml |
| 153 | 156 | Sitemap: https://www.wsj.com/wsjsitemaps/wsj_recipes.xml |
| 154 | 157 | Sitemap: https://www.wsj.com/sitemap_topic_collections.xml |
| 158 | + | |
| 159 | + |