NeuralCrawl

Reuters / robots.txt snapshot

← back to reuters.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 3700 bytes · sha256 e95fb3c36e6be7f4 · raw

final URL: https://www.reuters.com/robots.txt

1User-agent: AASA-Bot
2User-agent: ADmantX
3User-agent: AdsBot-Google
4User-agent: AdsBot-Google-Mobile
5User-agent: AmazonAdBot
6User-agent: Amzn-User
7User-agent: Applebot
8User-agent: AppleNewsBot
9User-agent: Bingbot
10User-agent: BingPreview
11User-agent: BrightEdge
12User-agent: BrightEdgeOnCrawl
13User-agent: CensysInspect
14User-agent: ChatGPT-User
15User-agent: Cision
16User-agent: Clickagy
17User-agent: Concert
18User-agent: ContextualBot
19User-agent: CriteoBot
20User-agent: DatadogSynthetics
21User-agent: datadome-pageprotect-scanner
22User-agent: Discordbot
23User-agent: doubleverify
24User-agent: DuckDuckBot
25User-agent: ElevenlabsBot
26User-agent: Embedly
27User-agent: facebookexternalhit
28User-agent: Googlebot
29User-agent: Googlebot Smartphone
30User-agent: Googlebot-News
31User-agent: Google-Display-Ads-Bot
32User-agent: Google-InspectionTool
33User-agent: GoogleOther
34User-agent: Google-Read-Aloud
35User-agent: Google-Safety
36User-agent: Google-Site-Verification
37User-agent: GTmetrix
38User-agent: GumGumBot
39User-agent: ias_crawler
40User-agent: Iframely
41User-agent: leiki
42User-agent: LinkedInBot
43User-agent: LinkTiger
44User-agent: Mantisbot
45User-agent: Mediapartners-Google
46User-agent: meta-externalads
47User-agent: meta-webindexer
48User-agent: MicrosoftPreview
49User-agent: MJ12bot
50User-agent: Moreover
51User-agent: msnbot
52User-agent: NFBNewslineRobot
53User-agent: OAI-SearchBot
54User-agent: Oncrawl
55User-agent: Opebot-v
56User-agent: Opoint
57User-agent: Optimizer
58User-agent: outbrain
59User-agent: Pinterestbot
60User-agent: Prerender
61User-agent: Proofpoint
62User-agent: proximic
63User-agent: PubMatic Crawler Bot
64User-agent: Quantcastbot
65User-agent: Qwantbot
66User-agent: Reuters SEO Screaming Frog Spider 007
67User-agent: Reuters-NAUWI
68User-agent: Scom-Crawler-For-Reuters
69User-agent: SinceraSyntheticUser
70User-agent: Slurp
71User-agent: SmartologyBot
72User-agent: snews
73User-agent: SocialFlow
74User-agent: StatusCake
75User-agent: Storebot-Google
76User-agent: Stripebot
77User-agent: TTD-Content
78User-agent: Twitterbot
79User-agent: URLDefense
80User-agent: Verity
81User-agent: vuln_scan_by_trustedsite_com_halo_security
82User-agent: WISEbot
83User-agent: Xenu Link Sleuth
84User-agent: Yahoo Link Preview
85User-agent: Yahoo! JAPAN
86User-agent: YahooMailProxy
87Disallow: /finance/stocks/option
88Disallow: /finance/stocks/financialHighlights
89Disallow: /search
90Disallow: /site-search/
91Disallow: /beta
92Disallow: /designtech
93Disallow: /featured-optimize
94Disallow: /energy-test
95Disallow: /article/beta
96Disallow: /sponsored/previewcampaign
97Disallow: /sponsored/previewarticle
98Disallow: /test/
99Disallow: /news/archive/commentary
100Disallow: /brandfeatures/venture-capital
101Disallow: /assets/siteindex
102Disallow: /article/api/
103Disallow: /practical-law-the-journal/search/
104Disallow: /pf/api/
105Disallow: /fr/
106Disallow: /it/
107Disallow: /es/
108Disallow: /pt/
109Disallow: /de/
110Disallow: /latam/
111Disallow: /account/subscribe/payment/
112
113# Block all other bots
114User-agent: *
115Allow: /plus/
116Disallow: /
117
118SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml
119SITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml
120SITEMAP: https://www.reuters.com/plus/sitemap-index.xml
121SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml
122SITEMAP: https://www.reuters.com/graphics/sitemap.xml
123SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/pictures/?outputType=xml
124SITEMAP: https://www.reuters.com/static/video-sitemap/us/sitemap_video_index.xml
125SITEMAP: https://www.reuters.com/arc/outboundfeeds/topic-sitemap/?outputType=xml
126SITEMAP: https://www.reuters.com/arc/outboundfeeds/author-sitemap/?outputType=xml
127SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml