NeuralCrawl

Financial Times / robots.txt snapshot

← back to ft.com · fetched 2026-06-20T01:10:30Z (17h ago) · HTTP 200 · 1557 bytes · sha256 5bca28f815bc2e73 · raw

final URL: https://www.ft.com/robots.txt

1# All use of FT content is subject to the Terms & Conditions and
2# Copyright Policy set out on FT.com
3
4# We expressly prohibit any use of our content or data for any
5# machine learning or artificial intelligence purposes, including
6# the training or development of AI technologies or machine learning
7# language models.
8#
9# For any licensing queries please contact [email protected]
10
11Sitemap: https://www.ft.com/sitemaps/index.xml
12
13User-agent: *
14Disallow: /__
15Disallow: /_fs-ch
16Disallow: /article/e63d8640-eeb3-4473-b0ad-aa6aec16d9fe
17Disallow: /buy/
18Disallow: /cancel/
19Disallow: /join/
20Disallow: /login
21Disallow: /myft/
22Disallow: /offline/
23Disallow: /page-resources
24Disallow: /preferences/manage-cookies
25Disallow: /register/
26Disallow: /search
27Disallow: /signup
28Allow: /__origami/
29Allow: /__assets/
30Crawl-Delay: 1
31
32User-agent: CCBot
33Disallow: /
34
35User-agent: Google-Extended
36Disallow: /
37Allow: /*?syn-25a6b1a6=1
38
39User-agent: GoogleOther
40Disallow: /
41
42User-agent: anthropic-ai
43Disallow: /
44
45User-agent: omgili
46Disallow: /
47
48User-agent: omgilibot
49Disallow: /
50
51User-agent: Bytespider
52Disallow: /
53
54User-agent: cohere-ai
55Disallow: /
56
57User-agent: magpie-crawler
58Disallow: /
59
60User-agent: NewsNow
61Disallow: /
62
63User-agent: PerplexityBot
64Disallow: /
65
66User-agent: Perplexity-User
67Disallow: /
68
69User-agent: Claude-Web
70Disallow: /
71
72User-agent: ClaudeBot
73Disallow: /
74
75User-agent: Diffbot
76Disallow: /
77
78User-agent: FacebookBot
79Disallow: /
80
81User-agent: Applebot-Extended
82Disallow: /
83
84User-agent: meta-externalagent
85Disallow: /
86
87User-agent: Meta-ExternalFetcher
88Disallow: /