NeuralCrawl

Reuters / robots.txt snapshot

← back to reuters.com · fetched 2026-09-15T14:34:35Z (3d ago) · HTTP 200 · 4425 bytes · sha256 bb103befcbf8abf3 · raw

final URL: https://www.reuters.com/robots.txt

1# NOTICE: Collection of content, data and/or information from reuters.com through
2# automated means is prohibited unless you have prior written consent from Reuters
3# and may only be conducted for the purposes explicitly described in such permission.
4#
5# The terms and conditions for reuters.com may be found at 
6# https://www.reuters.com/info-pages/terms-of-use/
7#
8# If you would like to seek permission to license the content and/or other materials
9# of Reuters, please contact https://reutersagency.com/contact-us/enquire/
10
11User-agent: AASA-Bot
12User-agent: ADmantX
13User-agent: AdmiralBot
14User-agent: AdsBot-Google
15User-agent: AdsBot-Google-Mobile
16User-agent: AmazonAdBot
17User-agent: Amzn-User
18User-agent: Applebot
19User-agent: AppleNewsBot
20User-agent: Bingbot
21User-agent: BingPreview
22User-agent: BrightEdge
23User-agent: BrightEdgeOnCrawl
24User-agent: CensysInspect
25User-agent: ChatGPT-User
26User-agent: Cision
27User-agent: Clickagy
28User-agent: Concert
29User-agent: ContextualBot
30User-agent: CriteoBot
31User-agent: DatadogSynthetics
32User-agent: datadome-pageprotect-scanner
33User-agent: Discordbot
34User-agent: doubleverify
35User-agent: ElevenlabsBot
36User-agent: Embedly
37User-agent: facebookexternalhit
38User-agent: Googlebot
39User-agent: Googlebot Smartphone
40User-agent: Googlebot-News
41User-agent: Google-Display-Ads-Bot
42User-agent: Google-InspectionTool
43User-agent: GoogleOther
44User-agent: Google-Read-Aloud
45User-agent: Google-Safety
46User-agent: Google-Site-Verification
47User-agent: GTmetrix
48User-agent: GumGumBot
49User-agent: ias_crawler
50User-agent: Iframely
51User-agent: leiki
52User-agent: LinkedInBot
53User-agent: LinkTiger
54User-agent: Mantisbot
55User-agent: Mediapartners-Google
56User-agent: meta-externalads
57User-agent: meta-webindexer
58User-agent: MicrosoftPreview
59User-agent: MJ12bot
60User-agent: Moreover
61User-agent: msnbot
62User-agent: NFBNewslineRobot
63User-agent: OAI-SearchBot
64User-agent: Oncrawl
65User-agent: Opebot-v
66User-agent: Opoint
67User-agent: Optimizer
68User-agent: outbrain
69User-agent: Pinterestbot
70User-agent: Prerender
71User-agent: Proofpoint
72User-agent: proximic
73User-agent: PubMatic Crawler Bot
74User-agent: Quantcastbot
75User-agent: Qwantbot
76User-agent: Reuters SEO Screaming Frog Spider 007
77User-agent: Reuters-NAUWI
78User-agent: Scom-Crawler-For-Reuters
79User-agent: SinceraSyntheticUser
80User-agent: Slurp
81User-agent: SmartologyBot
82User-agent: snews
83User-agent: SocialFlow
84User-agent: StatusCake
85User-agent: Storebot-Google
86User-agent: Stripebot
87User-agent: TTD-Content
88User-agent: Twitterbot
89User-agent: URLDefense
90User-agent: Verity
91User-agent: vuln_scan_by_trustedsite_com_halo_security
92User-agent: WISEbot
93User-agent: Xenu Link Sleuth
94User-agent: Yahoo Link Preview
95User-agent: Yahoo! JAPAN
96User-agent: YahooMailProxy
97Disallow: /finance/stocks/option
98Disallow: /finance/stocks/financialHighlights
99Disallow: /search
100Disallow: /site-search/
101Disallow: /beta
102Disallow: /designtech
103Disallow: /featured-optimize
104Disallow: /energy-test
105Disallow: /article/beta
106Disallow: /sponsored/previewcampaign
107Disallow: /sponsored/previewarticle
108Disallow: /test/
109Disallow: /news/archive/commentary
110Disallow: /brandfeatures/venture-capital
111Disallow: /assets/siteindex
112Disallow: /article/api/
113Disallow: /practical-law-the-journal/search/
114Disallow: /pf/api/
115Disallow: /fr/
116Disallow: /it/
117Disallow: /es/
118Disallow: /pt/
119Disallow: /de/
120Disallow: /latam/
121Disallow: /account/subscribe/payment
122
123User-agent: Applebot-Extended
124Disallow: /
125
126# Block all other bots
127User-agent: *
128Allow: /plus/
129Disallow: /
130
131SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml
132SITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml
133SITEMAP: https://www.reuters.com/plus/sitemap-index.xml
134SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml
135SITEMAP: https://www.reuters.com/graphics/sitemap.xml
136SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/pictures/?outputType=xml
137SITEMAP: https://www.reuters.com/static/video-sitemap/us/sitemap_video_index.xml
138SITEMAP: https://www.reuters.com/arc/outboundfeeds/topic-sitemap/?outputType=xml
139SITEMAP: https://www.reuters.com/arc/outboundfeeds/author-sitemap/?outputType=xml
140SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml
141SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml
142SITEMAP: https://www.reuters.com/service/archive-sitemap/sitemap.xml