NeuralCrawl

Arrow Electronics / robots.txt snapshot

← back to arrow.com · fetched 2026-10-13T05:44:23Z (25d ago) · HTTP 200 · 1622 bytes · sha256 ece32979f075eee5 · raw

final URL: https://www.arrow.com/robots.txt

1User-agent: *
2Disallow: */cart$
3Disallow: /checkout
4Disallow: */sitecore/
5
6Disallow: /akamai/akamai-sure-route.html
7
8Disallow: */reference-designs/search
9Disallow: */design-center/my-projects
10Disallow: /technical-content/
11
12Disallow: */datasheets/
13Disallow: */login*
14
15Disallow: /~/media/technical-content/
16Disallow: */my-account
17
18Disallow: */indiegogo/te-connectivity
19
20Disallow: /marketingtracking/
21Disallow: /_bm/
22Disallow: /samtec-product-selector
23Disallow: /apps/samtec/configurator/sfm
24
25Disallow: /en/register
26Disallow: /es/register
27Disallow: /de/register
28Disallow: /fr/register
29Disallow: /it/register
30Disallow: /ko/register
31Disallow: /zh/register
32Disallow: /ja/register
33
34Disallow: */shellpages
35Disallow: */errors
36
37User-agent: anthropic-ai
38Disallow: /
39Allow: /ais/*
40Allow: /globalecs/*
41User-agent: ClaudeBot
42Disallow: /
43Allow: /ais/*
44Allow: /globalecs/*
45
46User-agent: Google-Extended
47Disallow: /
48Allow: /ais/*
49Allow: /globalecs/*
50
51User-agent: GPTBot
52Disallow: /
53Allow: /ais/*
54Allow: /globalecs/*
55
56User-agent: ChatGPT-User
57Disallow: /
58Allow: /ais/*
59Allow: /globalecs/*
60
61User-agent: CCBot
62Disallow: /
63Allow: /ais/*
64Allow: /globalecs/*
65
66User-agent: PerplexityBot
67Disallow: /
68Allow: /ais/*
69Allow: /globalecs/*
70
71Sitemap: https://www.arrow.com/sitemap.xml
72
73Sitemap: https://www.arrow.com/commerce/sitemap/products/sitemap-products-index.xml
74
75Sitemap: https://www.arrow.com/commerce/sitemap/categories/sitemap-categories-index.xml
76
77Sitemap: https://www.arrow.com/commerce/sitemap/manufacturers/sitemap-manufacturers-index.xml
78
79Sitemap: https://www.arrow.com/commerce/sitemap/referencedesigns/sitemap-referencedesign-index.xml