NeuralCrawl

Insight Enterprises / robots.txt

← back to insight.com · change detected 1mo ago (2026-09-22T04:22:28Z) · modified +91 −19

What changed

Diff old (2026-08-28T02:24:20Z) → new (2026-09-22T04:22:28Z)

@@ -1,44 +1,116 @@
11  # Robots.txt for Insight.com
2- # Use specialized blocks only if rules differ from the global policy.
2+ # Global policy first; specialized blocks only where rules differ.
3+ # Precedence note: Google/Bing resolve conflicts by longest matching rule
4+ # (Allow wins ties). Rules below are ordered for readability, not precedence.
35 
46  User-agent: *
5- # 1. ALLOW RULES (Specific parameters & paths)
6- Allow: /*?qtype=
7- Allow: /*?pq=
8- Allow: /*?identifier=shopping
9- Allow: /*?partnermessage
7+
8+ # 1. ALLOW RULES (Rendering Assets, Core Paths & Query Parameters)
109  Allow: /insightweb/*.css$
10+ Allow: /insightweb/*.css?
11+ Allow: /insightweb/*.js$
12+ Allow: /insightweb/*.js?
13+ Allow: /insightweb/*.svg$
14+ Allow: /insightweb/*.svg?
15+ Allow: /insightweb/*.woff$
16+ Allow: /insightweb/*.woff2$
17+
18+ Allow: /shop/product/
1119  Allow: /*/shop/product/
1220 
13- # 2. DISALLOW RULES (System folders and global parameters)
14- Disallow: /*?*
15- Disallow: /*/search*.html
21+ # Whitelisted query parameters (these outrank the product query-string
22+ # disallows below because they are longer, more specific rules)
23+ Allow: /*qtype=
24+ Allow: /*pq=
25+ Allow: /*identifier=shopping
26+ Allow: /*partnermessage=
27+
28+ # 2. DISALLOW RULES (System folders, Internal DAM & Query Strings)
29+ # Blocks all parameterized query URLs globally (including /search.html?q=...)
30+ Disallow: /*?
31+
32+ # Target all localized search pages specifically (with or without parameters)
33+ Disallow: /search.html*
34+ Disallow: /*/search.html*
35+
36+ # Prevent /shop/product/ allows (above) from opening ALL parameterized
37+ # product URLs (faceted nav, sort orders, tracking params).
38+ # Whitelisted params still crawl via the longer Allow rules above.
39+ Disallow: /shop/product/*?
40+ Disallow: /*/shop/product/*?
41+
42+ # System Folders
1643  Disallow: /insightweb/
1744  Disallow: /flytrap/
45+
46+ # AEM Content & DAM Paths
47+ Disallow: /content/dam/insight/
1848  Disallow: /content/dam/insight-web/*/solutions/service-provider/microsite/assets/
1949  Disallow: /content/dam/insight-web/*/pdfs/
20- Disallow: /content/dam/insight/
2150  Disallow: /content/dam/global/*/pdfs/
22- Disallow: /content/insight-web/*/help/*
23- Disallow: /content/insight-web/*/client/*
24- Disallow: /content/insight-web/*/Sandbox/*
25- Disallow: /content/insight-web/*/sandbox/*
51+ Disallow: /content/dam/global/media/ppt/
52+ Disallow: /content/dam/global/media/zip/
53+ Disallow: /content/dam/global/*/media/
54+ Disallow: /content/dam/global/*/clientdocs/
55+
56+ # Staging, Help & Sandbox Folders (Wildcard ending blocks both slash and non-slash paths)
57+ Disallow: /content/insight-web/*/help*
58+ Disallow: /content/insight-web/*/client*
59+ Disallow: /content/insight-web/*/Sandbox*
60+ Disallow: /content/insight-web/*/sandbox*
2661 
2762  ############################
28- # BLOCKED CRAWLERS
63+ # AI TRAINING CRAWLERS - BLOCKED
64+ # Policy: block model training; remain visible in AI search/answers.
65+ # Search & user-triggered agents (OAI-SearchBot, ChatGPT-User,
66+ # Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User)
67+ # are intentionally NOT blocked - they drive citations and referrals.
2968  ############################
69+
70+ # OpenAI - training
71+ User-agent: GPTBot
72+ Disallow: /
73+
74+ # Anthropic - training
75+ User-agent: ClaudeBot
76+ Disallow: /
77+
78+ # Google - Gemini training/grounding opt-out (does not affect Google Search)
79+ User-agent: Google-Extended
80+ Disallow: /
81+
82+ # Apple - Apple Intelligence training opt-out (does not affect Applebot/Siri search)
83+ User-agent: Applebot-Extended
84+ Disallow: /
85+
86+ # Meta - training (current token)
87+ User-agent: meta-externalagent
88+ Disallow: /
89+
90+ # Meta - training (legacy token, kept for coverage)
91+ User-agent: FacebookBot
92+ Disallow: /
93+
94+ # Common Crawl - training corpora
3095  User-agent: CCBot
3196  Disallow: /
3297 
33- User-agent: FacebookBot
98+ # Amazon - training
99+ User-agent: Amazonbot
34100  Disallow: /
35101 
36- User-agent: NeevaAI
37- Disallow: /
38-
102+ # ByteDance - training. Known to ignore robots.txt;
103+ # enforce at CDN/WAF level as well.
39104  User-agent: Bytespider
40105  Disallow: /
41106 
107+ # Perplexity - training/crawling (excludes user-triggered search citations)
108+ User-agent: Perplexity
109+ Disallow: /
110+
111+ ############################
112+ # COMMERCIAL SCRAPERS - BLOCKED
113+ ############################
42114  User-agent: Firecrawl
43115  Disallow: /
44116