NeuralCrawl

First American Financial / robots.txt snapshot

← back to firstam.com · fetched 2026-11-05T07:45:14Z (2d ago) · HTTP 200 · 2676 bytes · sha256 3073d1e371afb327 · raw

final URL: https://firstam.com/robots.txt

1# As a condition of accessing this website, you agree to abide by the following
2# content signals:
3
4# (a) If a Content-Signal = yes, you may collect content for the corresponding
5# use.
6# (b) If a Content-Signal = no, you may not collect content for the
7# corresponding use.
8# (c) If the website operator does not include a Content-Signal for a
9# corresponding use, the website operator neither grants nor restricts
10# permission via Content-Signal with respect to the corresponding use.
11
12# The content signals and their meanings are:
13
14# search: building a search index and providing search results (e.g., returning
15# hyperlinks and short excerpts from your website's contents). Search does not
16# include providing AI-generated search summaries.
17# ai-input: inputting content into one or more AI models (e.g., retrieval
18# augmented generation, grounding, or other real-time taking of content for
19# generative AI search answers).
20# ai-train: training or fine-tuning AI models.
21# use: how AI systems may consume the content (immediate, reference, or full).
22
23# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
26
27# BEGIN Cloudflare Managed content
28
29
30User-agent: *
31Content-Signal: search=yes,ai-train=no,use=reference
32Allow: /
33
34# Training crawlers
35User-agent: Amazonbot
36Disallow: /
37
38User-agent: Applebot-Extended
39Disallow: /
40
41User-agent: Bytespider
42Disallow: /
43
44User-agent: CCBot
45Disallow: /
46
47User-agent: ClaudeBot
48Disallow: /
49
50User-agent: Diffbot
51Disallow: /
52
53User-agent: Google-Extended
54Disallow: /
55
56User-agent: GPTBot
57Disallow: /
58
59User-agent: omgili
60Disallow: /
61
62User-agent: anthropic-ai
63Disallow: /
64
65User-agent: Claude-Web
66Disallow: /
67
68User-agent: cohere-ai
69Disallow: /
70
71User-agent: MistralAI-Training
72Disallow: /
73
74User-agent: meta-externalagent
75Disallow: /
76
77User-agent: GoogleOther
78Disallow: /
79
80User-agent: Baiduspider
81Disallow: /
82
83User-agent: PetalBot
84Disallow: /
85
86User-agent: AwarioSmartBot
87User-agent: AwarioRssBot
88Disallow: /
89
90User-agent: Google-CloudVertexBot
91Disallow: /
92
93User-agent: QualifiedBot
94Disallow: /
95
96User-agent: Cotoyogi
97Disallow: /
98
99User-agent: ICC-Crawler
100Disallow: /
101
102User-agent: atlassian-bot
103Disallow: /
104
105User-agent: FishBot
106Disallow: /
107
108User-agent: BorderxBot
109Disallow: /
110
111User-agent: NavuBot
112Disallow: /
113
114User-agent: SemrushBot-SWA
115Disallow: /
116
117User-agent: WARDBot
118Disallow: /
119
120User-agent: magpie-crawler
121Disallow: /
122
123User-agent: KimiBot
124Disallow: /
125
126User-agent: CitibotSiteCrawler
127Disallow: /
128
129# END Cloudflare Managed Content
130