NeuralCrawl

Reuters / robots.txt snapshot

← back to reuters.com · fetched 2026-08-17T11:01:05Z (1mo ago) · HTTP 200 · 3774 bytes · sha256 fc3e0e339380271d · raw

final URL: https://www.reuters.com/robots.txt

1User-agent: AASA-Bot
2User-agent: ADmantX
3User-agent: AdsBot-Google
4User-agent: AdsBot-Google-Mobile
5User-agent: AmazonAdBot
6User-agent: Amzn-User
7User-agent: Applebot
8User-agent: AppleNewsBot
9User-agent: Bingbot
10User-agent: BingPreview
11User-agent: BrightEdge
12User-agent: BrightEdgeOnCrawl
13User-agent: CensysInspect
14User-agent: ChatGPT-User
15User-agent: Cision
16User-agent: Clickagy
17User-agent: Concert
18User-agent: ContextualBot
19User-agent: CriteoBot
20User-agent: DatadogSynthetics
21User-agent: datadome-pageprotect-scanner
22User-agent: Discordbot
23User-agent: doubleverify
24User-agent: ElevenlabsBot
25User-agent: Embedly
26User-agent: facebookexternalhit
27User-agent: Googlebot
28User-agent: Googlebot Smartphone
29User-agent: Googlebot-News
30User-agent: Google-Display-Ads-Bot
31User-agent: Google-InspectionTool
32User-agent: GoogleOther
33User-agent: Google-Read-Aloud
34User-agent: Google-Safety
35User-agent: Google-Site-Verification
36User-agent: GTmetrix
37User-agent: GumGumBot
38User-agent: ias_crawler
39User-agent: Iframely
40User-agent: leiki
41User-agent: LinkedInBot
42User-agent: LinkTiger
43User-agent: Mantisbot
44User-agent: Mediapartners-Google
45User-agent: meta-externalads
46User-agent: meta-webindexer
47User-agent: MicrosoftPreview
48User-agent: MJ12bot
49User-agent: Moreover
50User-agent: msnbot
51User-agent: NFBNewslineRobot
52User-agent: OAI-SearchBot
53User-agent: Oncrawl
54User-agent: Opebot-v
55User-agent: Opoint
56User-agent: Optimizer
57User-agent: outbrain
58User-agent: Pinterestbot
59User-agent: Prerender
60User-agent: Proofpoint
61User-agent: proximic
62User-agent: PubMatic Crawler Bot
63User-agent: Quantcastbot
64User-agent: Qwantbot
65User-agent: Reuters SEO Screaming Frog Spider 007
66User-agent: Reuters-NAUWI
67User-agent: Scom-Crawler-For-Reuters
68User-agent: SinceraSyntheticUser
69User-agent: Slurp
70User-agent: SmartologyBot
71User-agent: snews
72User-agent: SocialFlow
73User-agent: StatusCake
74User-agent: Storebot-Google
75User-agent: Stripebot
76User-agent: TTD-Content
77User-agent: Twitterbot
78User-agent: URLDefense
79User-agent: Verity
80User-agent: vuln_scan_by_trustedsite_com_halo_security
81User-agent: WISEbot
82User-agent: Xenu Link Sleuth
83User-agent: Yahoo Link Preview
84User-agent: Yahoo! JAPAN
85User-agent: YahooMailProxy
86Disallow: /finance/stocks/option
87Disallow: /finance/stocks/financialHighlights
88Disallow: /search
89Disallow: /site-search/
90Disallow: /beta
91Disallow: /designtech
92Disallow: /featured-optimize
93Disallow: /energy-test
94Disallow: /article/beta
95Disallow: /sponsored/previewcampaign
96Disallow: /sponsored/previewarticle
97Disallow: /test/
98Disallow: /news/archive/commentary
99Disallow: /brandfeatures/venture-capital
100Disallow: /assets/siteindex
101Disallow: /article/api/
102Disallow: /practical-law-the-journal/search/
103Disallow: /pf/api/
104Disallow: /fr/
105Disallow: /it/
106Disallow: /es/
107Disallow: /pt/
108Disallow: /de/
109Disallow: /latam/
110Disallow: /jp/
111Disallow: /account/subscribe/payment
112
113# Block all other bots
114User-agent: *
115Allow: /plus/
116Disallow: /
117
118SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml
119SITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml
120SITEMAP: https://www.reuters.com/plus/sitemap-index.xml
121SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml
122SITEMAP: https://www.reuters.com/graphics/sitemap.xml
123SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/pictures/?outputType=xml
124SITEMAP: https://www.reuters.com/static/video-sitemap/us/sitemap_video_index.xml
125SITEMAP: https://www.reuters.com/arc/outboundfeeds/topic-sitemap/?outputType=xml
126SITEMAP: https://www.reuters.com/arc/outboundfeeds/author-sitemap/?outputType=xml
127SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml
128SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml