NeuralCrawl

KeyCorp / robots.txt

← back to key.com · change detected 1mo ago (2026-08-14T10:10:35Z) · modified +128 −5

What changed

Diff old (2026-06-20T01:10:31Z) → new (2026-08-14T10:10:35Z)

@@ -1,15 +1,138 @@
1+ # ============================================================
2+ # key.com robots.txt
3+ # Last updated: August 2026
4+ # AI crawler governance added — see sections below
5+ # ============================================================
6+
7+
8+ # ============================================================
9+ # SECTION 1: GENERAL CRAWLERS (all bots not named below)
10+ # ============================================================
111  User-agent: *
212  Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
3- Disallow: /kco/images/2019_KBCM_saas_survey_final.pdf
4- Disallow: /content/dam/kco/documents/businesses___institutions/2018_kbcm_saas_survey_pdf.pdf
513  Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
614  Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
715  Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
8- Disallow: /content/dam/kco/stand-alone/kbcm_saas_suvey_9-22-1_2_pdf.pdf
9- Disallow: /content/dam/kco/documents/businesses___institutions/keynavigator_microsite/lockbox-faq.pdf
10- Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_ach_instruction.pdf
1116  Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
1217  Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
1318  Disallow: /content/dam/kco/internal-assets/
1419 
20+
21+ # ============================================================
22+ # SECTION 2: AI SEARCH & RETRIEVAL CRAWLERS (ALLOW)
23+ # These bots power AI search results and citations.
24+ # Allowing them keeps KeyBank visible in ChatGPT Search,
25+ # Perplexity, Google AI Overviews, and similar products.
26+ # ============================================================
27+
28+ # OpenAI — SearchGPT / ChatGPT Search
29+ User-agent: OAI-SearchBot
30+ Allow: /
31+
32+ # OpenAI — live user requests in ChatGPT
33+ User-agent: ChatGPT-User
34+ Allow: /
35+
36+ # Perplexity AI — search retrieval
37+ User-agent: PerplexityBot
38+ Allow: /
39+
40+ # Perplexity AI — live user requests
41+ User-agent: Perplexity-User
42+ Allow: /
43+
44+ # Anthropic — Claude search retrieval (legacy)
45+ User-agent: Claude-Web
46+ Allow: /
47+
48+
49+ # ============================================================
50+ # SECTION 3: AI TRAINING CRAWLERS
51+ # Public content allowed for brand presence in future models.
52+ # Proprietary research and internal assets are blocked.
53+ # ============================================================
54+
55+ # OpenAI — GPT model training
56+ User-agent: GPTBot
57+ Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
58+ Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
59+ Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
60+ Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
61+ Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
62+ Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
63+ Disallow: /content/dam/kco/internal-assets/
64+
65+ # Anthropic — Claude model training
66+ User-agent: ClaudeBot
67+ Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
68+ Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
69+ Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
70+ Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
71+ Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
72+ Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
73+ Disallow: /content/dam/kco/internal-assets/
74+
75+ # Anthropic — legacy training crawler
76+ User-agent: anthropic-ai
77+ Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
78+ Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
79+ Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
80+ Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
81+ Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
82+ Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
83+ Disallow: /content/dam/kco/internal-assets/
84+
85+ # Google — Gemini training (policy token only — no separate user agent string)
86+ User-agent: Google-Extended
87+ Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
88+ Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
89+ Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
90+ Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
91+ Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
92+ Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
93+ Disallow: /content/dam/kco/internal-assets/
94+
95+ # Meta — Llama model training
96+ User-agent: Meta-ExternalAgent
97+ Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
98+ Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
99+ Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
100+ Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
101+ Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
102+ Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
103+ Disallow: /content/dam/kco/internal-assets/
104+
105+ # Meta — Facebook AI training
106+ User-agent: FacebookBot
107+ Disallow: /kco/images/2017_KBCM_Technology_Group_Private_SaaS_Company_Survey.pdf
108+ Disallow: /content/dam/kco/documents/businesses___institutions/2024_kbcm_sapphire_saas_survey.pdf
109+ Disallow: /content/dam/kco/documents/businesses___institutions/kbcm_2023_sapphire_ventures_saas_survey.pdf
110+ Disallow: /content/dam/kco/documents/businesses___institutions/2022_kbcm_saas_survey_10-20-22_vF.pdf
111+ Disallow: /content/dam/kco/cf/disclosure/businesses-institutions/lockbox-processing-platform-updates.pdf
112+ Disallow: /content/dam/kco/documents/small_business/kbbo_pay_with_wire_user_guide.pdf
113+ Disallow: /content/dam/kco/internal-assets/
114+
115+
116+ # ============================================================
117+ # SECTION 4: AGGRESSIVE / NON-COMPLIANT SCRAPERS (BLOCK ALL)
118+ # These crawlers feed less-regulated pipelines and have a
119+ # poor track record of honoring robots.txt directives.
120+ # ============================================================
121+
122+ # ByteDance / TikTok AI training
123+ User-agent: Bytespider
124+ Disallow: /
125+
126+ # Common Crawl — bulk scraper used by many unvetted pipelines
127+ User-agent: CCBot
128+ Disallow: /
129+
130+ # Diffbot — commercial data extraction
131+ User-agent: Diffbot
132+ Disallow: /
133+
134+
135+ # ============================================================
136+ # SITEMAP
137+ # ============================================================
15138  Sitemap: https://www.key.com/about/seo/sitemap.xml