NeuralCrawl

Bloomberg / robots.txt snapshot

← back to bloomberg.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 6433 bytes · sha256 2a8fb822901ff7b3 · raw

final URL: https://www.bloomberg.com/robots.txt

1# Bot rules:
2# 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm.
3# 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law.
4# 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law.
5# If you can read this then you should apply here https://www.bloomberg.com/careers/
6User-agent: *
7Disallow: /polska
8Allow: /account/newsletters
9Disallow: /account/*
10Disallow: /tosv*.html
11Disallow: /search
12Disallow: /company/search/
13Disallow: /professional/search/
14Disallow: /impact/search/
15Disallow: /ux/search/
16Disallow: /wnwi/search/
17Disallow: /gei/search/
18Disallow: /impact/search/
19Disallow: /netzeropathfinders/search/
20Disallow: /notices/search/
21Disallow: /distribution/search/
22Disallow: /ukinnovators/search/
23Disallow: /latam/search/
24Disallow: /faq/search/
25Disallow: /tc/search/
26Disallow: /subscriptions/group/manage/
27Disallow: /preview/lineup
28Disallow: /preview/articles
29Disallow: /explore/
30Disallow: /press-releases/
31Disallow: /artemis/
32Disallow: /sessions-publisher/
33
34User-agent: Google-Extended
35Disallow: /
36Allow: /professional
37Allow: /company
38Allow: /latam
39Allow: /faq
40Allow: /tc
41
42User-agent: Mediapartners-Google
43Disallow: /about/careers
44Disallow: /about/careers/
45Disallow: /offlinemessage/
46Disallow: /apps/fbk
47Disallow: /bb/newsarchive/
48Disallow: /apps/news
49
50User-agent: Spinn3r
51Disallow: /podcasts/
52Disallow: /feed/podcast/
53Disallow: /bb/avfile/
54
55User-agent: Googlebot-News
56Disallow: /sponsor/
57Disallow: /news/sponsors/*
58Disallow: /news/terminal/*
59
60User-agent: Twitterbot
61Allow: /en/news/thp
62
63User-agent: GPTBot
64Disallow: /
65
66User-agent: CCBot
67Disallow: /
68
69User-agent: Google-Extended
70Disallow: /
71
72# Development Tools
73User-agent: Python-urllib
74Disallow: /
75User-agent: python-requests
76Disallow: /
77User-agent: Python-http-client
78Disallow: /
79
80# AI/LLM Bots
81User-agent: ChatGPT-User
82Disallow: /
83Allow: /professional
84Allow: /company
85Allow: /latam
86Allow: /faq
87Allow: /tc
88User-agent: GPTBot
89Disallow: /
90Allow: /professional
91Allow: /company
92Allow: /latam
93Allow: /faq
94Allow: /tc
95User-agent: Claude-Web
96Disallow: /
97Allow: /professional
98Allow: /company
99Allow: /latam
100Allow: /faq
101Allow: /tc
102User-agent: anthropic-ai
103Disallow: /
104Allow: /professional
105Allow: /company
106Allow: /latam
107Allow: /faq
108Allow: /tc
109User-agent: PerplexityBot
110Disallow: /
111Allow: /professional
112Allow: /company
113Allow: /latam
114Allow: /faq
115Allow: /tc
116User-agent: YouBot
117Disallow: /
118Allow: /professional
119Allow: /company
120Allow: /latam
121Allow: /faq
122Allow: /tc
123User-agent: ClaudeBot
124Disallow: /
125User-agent: Claude-User
126Disallow: /
127User-agent: Claude-SearchBot
128Disallow: /
129User-agent: OAI-SearchBot
130Disallow: /
131User-agent: DuckAssistBot
132Disallow: /
133User-agent: cohere-ai
134Disallow: /
135
136# Google Services
137User-agent: Google-Apps
138Disallow: /
139User-agent: Google-Apps-Script
140Disallow: /
141User-agent: AppEngine-Google
142Disallow: /
143User-agent: Google-Cloud
144Disallow: /
145User-agent: Google-CloudVertexBot
146Disallow: /
147
148# Major Company Bots
149User-agent: AmazonBot
150Disallow: /
151User-agent: AmazonAdBot
152Disallow: /
153User-agent: ByteSpider
154Disallow: /
155User-agent: Meta-ExternalAgent
156Disallow: /
157User-agent: Meta-ExternalFetcher
158Disallow: /
159User-agent: Meta-ExternalAgent-Image
160Disallow: /
161User-agent: Applebot-Extended
162Disallow: /
163User-agent: Bytespider-Image
164Disallow: /
165
166# Feed & Content Aggregators
167User-agent: Feedly
168Disallow: /
169User-agent: FeedlyBot
170Disallow: /
171User-agent: FeedlyApp
172Disallow: /
173User-agent: MWFeedParser
174Disallow: /
175
176# Analytics & Marketing
177User-agent: comscore
178Disallow: /
179User-agent: Comscore
180Disallow: /
181User-agent: HubSpot
182Disallow: /
183User-agent: hubspot
184Disallow: /
185User-agent: Criteo
186Disallow: /
187User-agent: criteo-bot
188Disallow: /
189User-agent: Peer39_Crawler
190Disallow: /
191User-agent: rogerbot
192Disallow: /
193User-agent: linkdexbot
194Disallow: /
195User-agent: NTENT
196Disallow: /
197User-agent: AndersPinkBot
198Disallow: /
199User-agent: IndeedBot
200Disallow: /
201
202# Search Engine Bots
203User-agent: MJ12bot
204Disallow: /
205User-agent: PetalBot
206Disallow: /
207User-agent: YisouSpider
208Disallow: /
209User-agent: 360Spider
210Disallow: /
211User-agent: Qwantify
212Disallow: /
213User-agent: ToutiaoSpider
214Disallow: /
215
216# Monitoring & Security
217User-agent: Screaming Frog SEO Spider
218Disallow: /
219Allow: /professional
220Allow: /company
221Allow: /latam
222User-agent: PRTG
223Disallow: /
224User-agent: FreshpingBot
225Disallow: /
226User-agent: Panopta
227Disallow: /
228User-agent: DatadogSynthetics
229Disallow: /
230User-agent: Rackspace
231Disallow: /
232User-agent: censys
233Disallow: /
234User-agent: burp
235Disallow: /
236User-agent: Burp
237Disallow: /
238User-agent: check_http
239Disallow: /
240User-agent: DotcomMonitor
241Disallow: /
242User-agent: WatchSumo
243Disallow: /
244User-agent: WormlyBot
245Disallow: /
246User-agent: CalibreBot
247Disallow: /
248User-agent: AudistoBot
249Disallow: /
250
251# Social & Content
252User-agent: Hatena
253Disallow: /
254User-agent: Hatena-Bookmark
255Disallow: /
256User-agent: EveryoneSocialBot
257Disallow: /
258User-agent: Diffbot
259Disallow: /
260User-agent: CCBot
261Disallow: /
262User-agent: Netvibes
263Disallow: /
264User-agent: WebCEO
265Disallow: /
266User-agent: Postano
267Disallow: /
268User-agent: RebelMouse
269Disallow: /
270User-agent: Muck-Rack
271Disallow: /
272User-agent: InstapaperViewer
273Disallow: /
274User-agent: Twurly
275Disallow: /
276User-agent: LivelapBot
277Disallow: /
278User-agent: DatagnionBot
279Disallow: /
280User-agent: Linespider
281Disallow: /
282User-agent: Discourse
283Disallow: /
284
285# Potentially Malicious
286User-agent: Medusa
287Disallow: /
288User-agent: pingback
289Disallow: /
290User-agent: WordPress
291Disallow: /
292User-agent: wp_ping
293Disallow: /
294User-agent: MauiBot
295Disallow: /
296User-agent: ltx71
297Disallow: /
298User-agent: WeSEE
299Disallow: /
300User-agent: halebot
301Disallow: /
302User-agent: BrightBot
303Disallow: /
304
305# Gaming/3D
306User-agent: UnityPlayer
307Disallow: /
308
309# Sitemaps app
310Sitemap: https://www.bloomberg.com/sitemaps/news/index.xml
311Sitemap: https://www.bloomberg.com/sitemaps/news/latest.xml
312Sitemap: https://www.bloomberg.com/sitemaps/collections/index.xml
313Sitemap: https://www.bloomberg.com/sitemaps/media/video/index.xml
314Sitemap: https://www.bloomberg.com/sitemaps/media/audio/index.xml
315Sitemap: https://www.bloomberg.com/sitemaps/people/profiles/index.xml
316Sitemap: https://www.bloomberg.com/sitemaps/companies/public-company/index.xml
317Sitemap: https://www.bloomberg.com/sitemaps/companies/private-company/index.xml
318Sitemap: https://www.bloomberg.com/sitemaps/securites/index.xml
319
320# Billionaires, owned by graphics
321Sitemap: https://www.bloomberg.com/billionaires/sitemap.xml