NeuralCrawl

Insight Enterprises / robots.txt

← back to insight.com · change detected 22d ago (2026-08-27T20:19:04Z) · modified +145 −8

What changed

Diff old (2026-06-20T01:10:31Z) → new (2026-08-27T20:19:04Z)

@@ -1,8 +1,18 @@
1- # Robots.txt for Insight.com
2- # Use specialized blocks only if rules differ from the global policy.
1+ # robots.txt for www.insight.com
2+ # Last updated: 2026-06-10
3+ #
4+ # Locale-agnostic design: Disallow rules use /* wildcards so they apply
5+ # across all locale prefixes (en_US, en_GB, de_DE, fr_FR, etc.) without
6+ # needing per-locale entries.
7+ #
8+ # Buy subdomain handled separately at buy.insight.com/robots.txt
39 
10+ # -----------------------------------------------
11+ # General crawlers
12+ # -----------------------------------------------
413  User-agent: *
5- # Allow specific parameters first
14+
15+ # Carve out specific query parameters that should remain crawlable
616  Allow: /*?qtype=
717  Allow: /*?pq=
818  Allow: /*?identifier=shopping
@@ -12,31 +22,158 @@
1222  Allow: /*/shop/product/
1323  Allow: /*%23*
1424 
15- # Block all other parameters and system folders
25+ # Block all other query parameters (catches session IDs, tracking,
26+ # print/preview flags, faceted search, etc. — the allows above take
27+ # precedence for whitelisted params)
1628  Disallow: /*?*
29+
30+ # Block internal search result pages (duplicate/low-value content)
1731  Disallow: /*/search*.html
32+
33+ # Block AEM app and system paths
1834  Disallow: /insightweb/
1935  Disallow: /flytrap/
36+
37+ # Block AEM content/dam asset paths (PDFs, raw DAM)
2038  Disallow: /content/dam/insight-web/*/solutions/service-provider/microsite/assets/
2139  Disallow: /content/dam/insight-web/*/pdfs/
2240  Disallow: /content/dam/insight/
2341  Disallow: /content/dam/global/*/pdfs/
42+
43+ # Block AEM content repository paths (help, client, sandbox)
2444  Disallow: /content/insight-web/*/help/*
2545  Disallow: /content/insight-web/*/client/*
2646  Disallow: /content/insight-web/*/Sandbox/*
2747  Disallow: /content/insight-web/*/sandbox/*
2848 
29- ############################
30- # BLOCKED CRAWLERS
31- ############################
49+ # Block internal/admin tooling
50+ Disallow: /admin/
51+ Disallow: /cms/
52+ Disallow: /api/
53+ Disallow: /internal/
54+
55+ # Block staging and test environments
56+ Disallow: /staging/
57+ Disallow: /test/
58+ Disallow: /uat/
59+ Disallow: /dev/
60+ Disallow: /sim/
61+ Disallow: /qa/
62+
63+ Crawl-delay: 5
64+
65+ # -----------------------------------------------
66+ # Blocked crawlers — AI training and data harvesting
67+ # -----------------------------------------------
68+ User-agent: GPTBot
69+ Disallow: /
70+
71+ User-agent: ChatGPT-User
72+ Disallow: /
73+
3274  User-agent: CCBot
75+ Disallow: /
76+
77+ User-agent: anthropic-ai
78+ Disallow: /
79+
80+ User-agent: Claude-Web
81+ Disallow: /
82+
83+ User-agent: cohere-ai
84+ Disallow: /
85+
86+ User-agent: Omgilibot
87+ Disallow: /
88+
3389  User-agent: FacebookBot
90+ Disallow: /
91+
92+ User-agent: Bytespider
93+ Disallow: /
94+
3495  User-agent: NeevaAI
35- User-agent: Bytespider
96+ Disallow: /
97+
3698  User-agent: Firecrawl
99+ Disallow: /
100+
37101  User-agent: Kadoa
102+ Disallow: /
103+
38104  User-agent: ImagesiftBot
39105  Disallow: /
40106 
107+ # -----------------------------------------------
108+ # Allowed referral/discovery crawlers
109+ # -----------------------------------------------
110+ User-agent: PerplexityBot
111+ Allow: /
41112 
113+ # -----------------------------------------------
114+ # Googlebot
115+ # -----------------------------------------------
116+ User-agent: Googlebot
117+
118+ Allow: /*?qtype=
119+ Allow: /*?pq=
120+ Allow: /*?identifier=shopping
121+ Allow: /*?partnermessage
122+ Allow: /*/shop/product/
123+
124+ Disallow: /*?*
125+ Disallow: /*/search*.html
126+ Disallow: /*/account/
127+ Disallow: /*/login
128+ Disallow: /*/logout
129+ Disallow: /*/register
130+ Disallow: /insightweb/
131+ Disallow: /flytrap/
132+ Disallow: /content/dam/insight-web/*/pdfs/
133+ Disallow: /content/dam/insight/
134+ Disallow: /content/insight-web/*/help/*
135+ Disallow: /content/insight-web/*/client/*
136+ Disallow: /content/insight-web/*/Sandbox/*
137+ Disallow: /content/insight-web/*/sandbox/*
138+ Disallow: /admin/
139+ Disallow: /api/
140+
141+ Crawl-delay: 1
142+
143+ # -----------------------------------------------
144+ # Googlebot-Image
145+ # -----------------------------------------------
146+ User-agent: Googlebot-Image
147+
148+ Disallow: /*/account/
149+ Disallow: /*/shop/cart
150+ Disallow: /*/shop/checkout
151+ Disallow: /content/dam/insight-web/*/pdfs/
152+ Disallow: /content/dam/global/*/pdfs/
153+
154+ # -----------------------------------------------
155+ # Bingbot
156+ # -----------------------------------------------
157+ User-agent: bingbot
158+
159+ Allow: /*?qtype=
160+ Allow: /*?pq=
161+ Allow: /*/shop/product/
162+
163+ Disallow: /*?*
164+ Disallow: /*/search*.html
165+ Disallow: /*/account/
166+ Disallow: /*/login
167+ Disallow: /insightweb/
168+ Disallow: /flytrap/
169+ Disallow: /content/dam/insight-web/*/pdfs/
170+ Disallow: /content/dam/insight/
171+ Disallow: /admin/
172+ Disallow: /api/
173+
174+ Crawl-delay: 2
175+
176+ # -----------------------------------------------
177+ # Sitemaps
178+ # -----------------------------------------------
42179  Sitemap: https://www.insight.com/sitemap.xml