NeuralCrawl

Insight Enterprises / robots.txt snapshot

← back to insight.com · fetched 2026-08-28T02:24:20Z (22d ago) · HTTP 200 · 1168 bytes · sha256 39f67c00bbcde2dc · raw

final URL: https://www.insight.com/robots.txt

1# Robots.txt for Insight.com
2# Use specialized blocks only if rules differ from the global policy.
3
4User-agent: *
5# 1. ALLOW RULES (Specific parameters & paths)
6Allow: /*?qtype=
7Allow: /*?pq=
8Allow: /*?identifier=shopping
9Allow: /*?partnermessage
10Allow: /insightweb/*.css$
11Allow: /*/shop/product/
12
13# 2. DISALLOW RULES (System folders and global parameters)
14Disallow: /*?*
15Disallow: /*/search*.html
16Disallow: /insightweb/
17Disallow: /flytrap/
18Disallow: /content/dam/insight-web/*/solutions/service-provider/microsite/assets/
19Disallow: /content/dam/insight-web/*/pdfs/
20Disallow: /content/dam/insight/
21Disallow: /content/dam/global/*/pdfs/
22Disallow: /content/insight-web/*/help/*
23Disallow: /content/insight-web/*/client/*
24Disallow: /content/insight-web/*/Sandbox/*
25Disallow: /content/insight-web/*/sandbox/*
26
27############################
28# BLOCKED CRAWLERS
29############################
30User-agent: CCBot
31Disallow: /
32
33User-agent: FacebookBot
34Disallow: /
35
36User-agent: NeevaAI
37Disallow: /
38
39User-agent: Bytespider
40Disallow: /
41
42User-agent: Firecrawl
43Disallow: /
44
45User-agent: Kadoa
46Disallow: /
47
48User-agent: ImagesiftBot
49Disallow: /
50
51Sitemap: https://www.insight.com/sitemap.xml