NeuralCrawl

Insight Enterprises / robots.txt snapshot

← back to insight.com · fetched 2026-06-20T01:10:31Z (17h ago) · HTTP 200 · 1096 bytes · sha256 86d490e4f4480992 · raw

final URL: https://www.insight.com/robots.txt

1# Robots.txt for Insight.com
2# Use specialized blocks only if rules differ from the global policy.
3
4User-agent: *
5# Allow specific parameters first
6Allow: /*?qtype=
7Allow: /*?pq=
8Allow: /*?identifier=shopping
9Allow: /*?partnermessage
10Allow: /insightweb/*.css$
11Allow: /*.html
12Allow: /*/shop/product/
13Allow: /*%23*
14
15# Block all other parameters and system folders
16Disallow: /*?*
17Disallow: /*/search*.html
18Disallow: /insightweb/
19Disallow: /flytrap/
20Disallow: /content/dam/insight-web/*/solutions/service-provider/microsite/assets/
21Disallow: /content/dam/insight-web/*/pdfs/
22Disallow: /content/dam/insight/
23Disallow: /content/dam/global/*/pdfs/
24Disallow: /content/insight-web/*/help/*
25Disallow: /content/insight-web/*/client/*
26Disallow: /content/insight-web/*/Sandbox/*
27Disallow: /content/insight-web/*/sandbox/*
28
29############################
30# BLOCKED CRAWLERS
31############################
32User-agent: CCBot
33User-agent: FacebookBot
34User-agent: NeevaAI
35User-agent: Bytespider
36User-agent: Firecrawl
37User-agent: Kadoa
38User-agent: ImagesiftBot
39Disallow: /
40
41
42Sitemap: https://www.insight.com/sitemap.xml