NeuralCrawl

First American Financial / robots.txt

← back to firstam.com · change detected 2d ago (2026-11-05T07:45:14Z) · modified +129 −5

What changed

Diff old (2026-09-21T22:21:50Z) → new (2026-11-05T07:45:14Z)

@@ -1,6 +1,130 @@
1+ # As a condition of accessing this website, you agree to abide by the following
2+ # content signals:
3+
4+ # (a) If a Content-Signal = yes, you may collect content for the corresponding
5+ # use.
6+ # (b) If a Content-Signal = no, you may not collect content for the
7+ # corresponding use.
8+ # (c) If the website operator does not include a Content-Signal for a
9+ # corresponding use, the website operator neither grants nor restricts
10+ # permission via Content-Signal with respect to the corresponding use.
11+
12+ # The content signals and their meanings are:
13+
14+ # search: building a search index and providing search results (e.g., returning
15+ # hyperlinks and short excerpts from your website's contents). Search does not
16+ # include providing AI-generated search summaries.
17+ # ai-input: inputting content into one or more AI models (e.g., retrieval
18+ # augmented generation, grounding, or other real-time taking of content for
19+ # generative AI search answers).
20+ # ai-train: training or fine-tuning AI models.
21+ # use: how AI systems may consume the content (immediate, reference, or full).
22+
23+ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24+ # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25+ # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
26+
27+ # BEGIN Cloudflare Managed content
28+
29+
130  User-agent: *
2- Disallow: /error/
3- Disallow: /falive/
4- Disallow: /title/error/
5- Disallow: /ownership/
6- Sitemap: https://www.firstam.com/sitemap.xml
31+ Content-Signal: search=yes,ai-train=no,use=reference
32+ Allow: /
33+
34+ # Training crawlers
35+ User-agent: Amazonbot
36+ Disallow: /
37+
38+ User-agent: Applebot-Extended
39+ Disallow: /
40+
41+ User-agent: Bytespider
42+ Disallow: /
43+
44+ User-agent: CCBot
45+ Disallow: /
46+
47+ User-agent: ClaudeBot
48+ Disallow: /
49+
50+ User-agent: Diffbot
51+ Disallow: /
52+
53+ User-agent: Google-Extended
54+ Disallow: /
55+
56+ User-agent: GPTBot
57+ Disallow: /
58+
59+ User-agent: omgili
60+ Disallow: /
61+
62+ User-agent: anthropic-ai
63+ Disallow: /
64+
65+ User-agent: Claude-Web
66+ Disallow: /
67+
68+ User-agent: cohere-ai
69+ Disallow: /
70+
71+ User-agent: MistralAI-Training
72+ Disallow: /
73+
74+ User-agent: meta-externalagent
75+ Disallow: /
76+
77+ User-agent: GoogleOther
78+ Disallow: /
79+
80+ User-agent: Baiduspider
81+ Disallow: /
82+
83+ User-agent: PetalBot
84+ Disallow: /
85+
86+ User-agent: AwarioSmartBot
87+ User-agent: AwarioRssBot
88+ Disallow: /
89+
90+ User-agent: Google-CloudVertexBot
91+ Disallow: /
92+
93+ User-agent: QualifiedBot
94+ Disallow: /
95+
96+ User-agent: Cotoyogi
97+ Disallow: /
98+
99+ User-agent: ICC-Crawler
100+ Disallow: /
101+
102+ User-agent: atlassian-bot
103+ Disallow: /
104+
105+ User-agent: FishBot
106+ Disallow: /
107+
108+ User-agent: BorderxBot
109+ Disallow: /
110+
111+ User-agent: NavuBot
112+ Disallow: /
113+
114+ User-agent: SemrushBot-SWA
115+ Disallow: /
116+
117+ User-agent: WARDBot
118+ Disallow: /
119+
120+ User-agent: magpie-crawler
121+ Disallow: /
122+
123+ User-agent: KimiBot
124+ Disallow: /
125+
126+ User-agent: CitibotSiteCrawler
127+ Disallow: /
128+
129+ # END Cloudflare Managed Content
130+