NeuralCrawl

The Wall Street Journal / robots.txt snapshot

← back to wsj.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 4245 bytes · sha256 c8a120e774d3d994 · raw

final URL: https://www.wsj.com/robots.txt

1# NOTICE: Collection of content and other data on https://www.wsj.com/ through
2# automated means is prohibited unless you have express written
3# permission from Dow Jones & Company, Inc. and may only be conducted for the
4# limited purpose contained in said permission.
5#
6# Dow Jones & Company, Inc. Terms of Use may be found at
7# https://www.dowjones.com/terms-of-use/
8#
9# If you would like to apply for permission to license the
10# intellectual property and/or other materials of Dow Jones & Company, Inc.’s
11# brands, please contact us via email at [email protected].
12
13User-agent: *
14Disallow: /
15
16User-agent: googlebot
17User-agent: googlebot-image
18User-agent: GoogleOther
19User-agent: Googlebot-Video
20User-agent: Google-InspectionTool
21User-agent: AdsBot-Google
22User-agent: AdsBot-Google-Mobile
23User-agent: AdsBot-Google-Mobile-Apps
24User-agent: Storebot-Google
25User-agent: google-read-aloud
26User-agent: mediapartners-google
27User-agent: bingbot
28User-agent: msnbot
29User-agent: bingpreview
30User-agent: slurp
31User-agent: yahoo
32User-agent: baiduspider
33User-agent: Pinterestbot
34User-agent: Yeti
35User-agent: MojeekBot
36User-agent: 360Spider
37User-agent: google-cloudvertexbot
38User-agent: duckduckbot
39User-agent: Applebot
40User-agent: flipboard
41User-agent: qwantbot
42User-agent: SeznamBot
43
44
45
46User-agent: proximic
47User-agent: admantx
48User-agent: thetradedesk
49User-agent: outbrain
50User-agent: ias_crawler
51User-agent: AmazonAdBot
52User-agent: pubmatic
53User-agent: smartologybot
54User-agent: parselybot
55
56
57User-agent: Screaming Frog SEO Spider
58User-agent: AhrefsBot
59User-agent: SemrushBot
60User-agent: SimilarWebBot
61User-agent: SISTRIX
62User-agent: botify
63User-agent: Chrome-Lighthouse
64
65
66User-agent: ChatGPT-User
67User-agent: GPTBot
68User-agent: OAI-SearchBot
69
70
71User-agent: facebookexternalhit
72User-agent: facebot
73User-agent: twitterbot
74User-agent: linkedinbot
75User-agent: snapchat
76User-agent: sentry
77User-agent: Iframely
78User-agent: Vocabtracker
79User-agent: EpvzCrawl6194680250
80User-agent: Citoid
81User-agent: ZoteroTranslationServer
82
83
84
85Allow: /
86
87User-agent: mediapartners-google
88Disallow: /
89Allow: /watchlist
90
91Disallow: /article_email/*
92Disallow: /user/*
93Disallow: /pdf/documents/*
94Disallow: /login/*
95Disallow: /acct/*
96Disallow: /msgcenter/*
97Disallow: /setup/*
98Disallow: /marketing/*
99Disallow: /public/article/*
100Disallow: /public/resources/documents/*
101Disallow: /public/search/
102Disallow: /public/search*
103Disallow: /search*
104Disallow: /public/page/wsj-x-marketing.html
105Disallow: /public/page/news-media-marketing.html
106Disallow: /public/page/0_0_WP_RT_MARKETING.html
107Disallow: /news/articles/SB2*
108Disallow: /news/articles/SB3*
109Disallow: /news/articles/SB4*
110Disallow: /articles/SB2*
111Disallow: /articles/SB3*
112Disallow: /articles/SB4*
113Disallow: /article/AP*
114Disallow: /article/BT-CO*
115Disallow: /article/DN-CO*
116Disallow: /article/PR-CO*
117Disallow: /article/HUG*
118Disallow: /video/search/*
119Disallow: /articles/BT-CO*
120Disallow: /articles/DN-CO*
121Disallow: /articles/PR-CO*
122Disallow: /news/articles/BT-CO*
123Disallow: /news/articles/DN-CO*
124Disallow: /news/articles/PR-CO*
125Disallow: /catchup/*
126Disallow: /articles/the-meaning-behind-juneteenth-11592413234
127Disallow: /emailservice/*
128Disallow: /emailsignup/*
129Disallow: /insetsrv/v1/*
130Disallow: /user/fpd/api/*
131Disallow: /Date(*
132Disallow: /auth/sso/proxy-login*
133Disallow: /client/
134
135# For Buyside Search Results
136Disallow: /buyside/search-results?*term=*
137# Don't crawl non-indexable sites
138Disallow: /*?type=mdc_*&id=*
139Disallow: /*?id=*&type=mdc_*
140Disallow: /market-data/quotes/*/options/*
141Disallow: /subscribe/?inttrackingCode=*
142Disallow: /subscribe/?template=*
143
144Sitemap: https://www.wsj.com/sitemap.xml
145Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml
146Sitemap: https://www.wsj.com/wsj_video_recent.xml
147Sitemap: https://www.wsj.com/sitemap_topics.xml
148Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml
149Sitemap: https://www.wsj.com/live_news_sitemap.xml
150Sitemap: https://www.wsj.com/authors_sitemap.xml
151Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml
152Sitemap: https://www.wsj.com/buyside/sitemap.xml
153Sitemap: https://www.wsj.com/wsj_quote_index_sitemap.xml
154Sitemap: https://www.wsj.com/wsjsitemaps/wsj_recipes.xml
155Sitemap: https://www.wsj.com/sitemap_topic_collections.xml