NeuralCrawl

Walmart / robots.txt snapshot

← back to walmart.com · fetched 2026-10-12T11:21:56Z (26d ago) · HTTP 200 · 3479 bytes · sha256 502a5efeb6a86722 · raw

final URL: https://www.walmart.com/robots.txt

1#Sitemaps-https
2
3Sitemap: https://www.walmart.com/sitemap_category.xml
4Sitemap: https://www.walmart.com/sitemap_store_main.xml
5
6Sitemap: https://www.walmart.com/help/sitemap_gm.xml
7Sitemap: https://www.walmart.com/sitemap_browse_fst.xml
8Sitemap: https://www.walmart.com/sitemap_store_dept.xml
9
10Sitemap: https://www.walmart.com/sitemap_tp_legacy.xml
11Sitemap: https://www.walmart.com/sitemap_tp_at.xml
12Sitemap: https://www.walmart.com/sitemap_tp_br.xml
13Sitemap: https://www.walmart.com/sitemap_tp_kp.xml
14
15Sitemap: https://www.walmart.com/sitemap_brp_02.xml
16Sitemap: https://www.walmart.com/sitemap_brp_03.xml
17Sitemap: https://www.walmart.com/sitemap_brp_04.xml
18Sitemap: https://www.walmart.com/sitemap_brp_05.xml
19Sitemap: https://www.walmart.com/sitemap_brp_06.xml
20Sitemap: https://www.walmart.com/sitemap_brp_07.xml
21Sitemap: https://www.walmart.com/sitemap_brp_08.xml
22Sitemap: https://www.walmart.com/sitemap_brp_09.xml
23
24Sitemap: https://www.walmart.com/sitemap_itp_01.xml
25Sitemap: https://www.walmart.com/sitemap_itp_02.xml
26Sitemap: https://www.walmart.com/sitemap_product_03.xml
27
28Sitemap: https://www.walmart.com/sitemap_rp_01.xml
29Sitemap: https://www.walmart.com/sitemap_rp_02.xml
30Sitemap: https://www.walmart.com/sitemap_rp_03.xml
31Sitemap: https://www.walmart.com/sitemap_rp_04.xml
32Sitemap: https://www.walmart.com/sitemap_rp_05.xml
33
34Sitemap: https://www.walmart.com/sitemap_auto_dir.xml
35
36Sitemap: https://www.walmart.com/sitemap_brndshp_brand.xml
37
38Sitemap: https://www.walmart.com/sitemap_sp.xml
39
40Sitemap: https://www.walmart.com/sitemap_hi_ip.xml
41Sitemap: https://www.walmart.com/sitemap_hi_tp.xml
42Sitemap: https://www.walmart.com/sitemap_hi_bp.xml
43Sitemap: https://www.walmart.com/sitemap_hi_mp.xml
44
45
46#Disallow select URLs
47User-agent: *
48Disallow: /0/
49Disallow: /55875582/walmart-us/catalog/
50Disallow: /account/
51Disallow: /api/
52Disallow: /collection/api/logger
53Disallow: /cp/-201
54Disallow: /cp/-302
55Disallow: /cp/-306
56Disallow: /cp/-309
57Disallow: /cp/-506
58Disallow: /cp/-509
59Disallow: /cp/api/logger
60Disallow: /cp/api/wpa
61Disallow: /cservice/
62Disallow: /cservice/ya_index.gsp
63Disallow: /electrode/api/logger
64Disallow: /email_collect/
65Disallow: /giftregistry/
66Disallow: /msp
67Disallow: /nonConfig/api/wpa
68Disallow: /popup_security.jsp
69Disallow: /product/electrode/api/logger
70Disallow: /product/electrode/api/wpa
71Allow: /reviews/product/
72Allow: /reviews/seller/
73Disallow: /search
74Disallow: /search/api/wpa
75Disallow: */api/wpa
76Disallow: */midas/*
77Disallow: /search/search-ng.do
78Disallow: /solutions/
79Disallow: /store/ajax/detail-navigation
80Disallow: /store/ajax/local-store
81Disallow: /store/ajax/preferred
82Disallow: /store/ajax/search
83Disallow: /store/ajax/visited-store
84Disallow: /store/category/
85Disallow: /store/electrode/api/fetch-coupons
86Disallow: /store/electrode/api/logger
87Disallow: /store/electrode/api/p13n
88Disallow: /store/electrode/api/search
89Disallow: /store/electrode/api/stores
90Allow: /store/finder
91Disallow: /store/popular_in_grade/
92Disallow: /storeLocator/
93Disallow: /storeLocator/ca_storefinder_results.do
94Disallow: /tealeaf
95Disallow: /topic/electrode/api/logger
96Disallow: /topic/electrode/api/wpa
97Disallow: /typeahead/
98Disallow: /wmflows/
99Disallow: */undefined/*
100Disallow: /store/*/search
101Disallow: /c/kp/*/page/*
102Disallow: /feeds/*
103Disallow: /orders
104Disallow: /es/
105Disallow: /swag/*
106Disallow: /dad/trk
107Disallow: /sp/*
108
109#Crawler specific settings
110User-agent: Adsbot-Google
111
112User-agent: Mediapartners-Google
113
114# slow down Yahoo
115User-agent: Slurp
116Crawl-delay: 5