NeuralCrawl

Western Digital / robots.txt snapshot

← back to westerndigital.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 697 bytes · sha256 1c6642e2d65f260e · raw

final URL: https://www.westerndigital.com/robots.txt

1User-agent: wdcbot # Western Digital
2Disallow: /content/dam/
3Allow: /content/dam/store/
4Allow: /content/dam/western-digital/
5Disallow: /etc/
6Allow: /etc/designs/
7Allow: /etc/clientlibs/
8Disallow: /libs/
9
10User-agent: OAI-SearchBot
11Disallow: /content/dam/
12Disallow: /etc/
13Disallow: /libs/
14Allow: /content/dam/store/
15Allow: /content/dam/western-digital/
16Allow: /etc/designs/
17Allow: /etc/clientlibs/
18
19User-agent: *
20Disallow: /content/dam/
21Disallow: /etc/
22Disallow: /libs/
23Allow: /content/dam/store/
24Allow: /content/dam/western-digital/
25Allow: /etc/designs/
26Allow: /etc/clientlibs/
27Disallow: /*?*filterBy*&*&*&*
28Disallow: /*?*filterBy*=*,*
29
30
31Sitemap: https://www.westerndigital.com/sitemap-index.xml