NeuralCrawl

Insight Enterprises / robots.txt snapshot

← back to insight.com · fetched 2026-09-22T04:22:28Z (1mo ago) · HTTP 200 · 3535 bytes · sha256 eba5a47083583b3e · raw

final URL: https://www.insight.com/robots.txt

1# Robots.txt for Insight.com
2# Global policy first; specialized blocks only where rules differ.
3# Precedence note: Google/Bing resolve conflicts by longest matching rule
4# (Allow wins ties). Rules below are ordered for readability, not precedence.
5
6User-agent: *
7
8# 1. ALLOW RULES (Rendering Assets, Core Paths & Query Parameters)
9Allow: /insightweb/*.css$
10Allow: /insightweb/*.css?
11Allow: /insightweb/*.js$
12Allow: /insightweb/*.js?
13Allow: /insightweb/*.svg$
14Allow: /insightweb/*.svg?
15Allow: /insightweb/*.woff$
16Allow: /insightweb/*.woff2$
17
18Allow: /shop/product/
19Allow: /*/shop/product/
20
21# Whitelisted query parameters (these outrank the product query-string
22# disallows below because they are longer, more specific rules)
23Allow: /*qtype=
24Allow: /*pq=
25Allow: /*identifier=shopping
26Allow: /*partnermessage=
27
28# 2. DISALLOW RULES (System folders, Internal DAM & Query Strings)
29# Blocks all parameterized query URLs globally (including /search.html?q=...)
30Disallow: /*?
31
32# Target all localized search pages specifically (with or without parameters)
33Disallow: /search.html*
34Disallow: /*/search.html*
35
36# Prevent /shop/product/ allows (above) from opening ALL parameterized
37# product URLs (faceted nav, sort orders, tracking params).
38# Whitelisted params still crawl via the longer Allow rules above.
39Disallow: /shop/product/*?
40Disallow: /*/shop/product/*?
41
42# System Folders
43Disallow: /insightweb/
44Disallow: /flytrap/
45
46# AEM Content & DAM Paths
47Disallow: /content/dam/insight/
48Disallow: /content/dam/insight-web/*/solutions/service-provider/microsite/assets/
49Disallow: /content/dam/insight-web/*/pdfs/
50Disallow: /content/dam/global/*/pdfs/
51Disallow: /content/dam/global/media/ppt/
52Disallow: /content/dam/global/media/zip/
53Disallow: /content/dam/global/*/media/
54Disallow: /content/dam/global/*/clientdocs/
55
56# Staging, Help & Sandbox Folders (Wildcard ending blocks both slash and non-slash paths)
57Disallow: /content/insight-web/*/help*
58Disallow: /content/insight-web/*/client*
59Disallow: /content/insight-web/*/Sandbox*
60Disallow: /content/insight-web/*/sandbox*
61
62############################
63# AI TRAINING CRAWLERS - BLOCKED
64# Policy: block model training; remain visible in AI search/answers.
65# Search & user-triggered agents (OAI-SearchBot, ChatGPT-User,
66# Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User)
67# are intentionally NOT blocked - they drive citations and referrals.
68############################
69
70# OpenAI - training
71User-agent: GPTBot
72Disallow: /
73
74# Anthropic - training
75User-agent: ClaudeBot
76Disallow: /
77
78# Google - Gemini training/grounding opt-out (does not affect Google Search)
79User-agent: Google-Extended
80Disallow: /
81
82# Apple - Apple Intelligence training opt-out (does not affect Applebot/Siri search)
83User-agent: Applebot-Extended
84Disallow: /
85
86# Meta - training (current token)
87User-agent: meta-externalagent
88Disallow: /
89
90# Meta - training (legacy token, kept for coverage)
91User-agent: FacebookBot
92Disallow: /
93
94# Common Crawl - training corpora
95User-agent: CCBot
96Disallow: /
97
98# Amazon - training
99User-agent: Amazonbot
100Disallow: /
101
102# ByteDance - training. Known to ignore robots.txt;
103# enforce at CDN/WAF level as well.
104User-agent: Bytespider
105Disallow: /
106
107# Perplexity - training/crawling (excludes user-triggered search citations)
108User-agent: Perplexity
109Disallow: /
110
111############################
112# COMMERCIAL SCRAPERS - BLOCKED
113############################
114User-agent: Firecrawl
115Disallow: /
116
117User-agent: Kadoa
118Disallow: /
119
120User-agent: ImagesiftBot
121Disallow: /
122
123Sitemap: https://www.insight.com/sitemap.xml