NeuralCrawl

Wiley / robots.txt snapshot

← back to wiley.com · fetched 2026-06-20T11:49:26Z (7h ago) · HTTP 200 · 3222 bytes · sha256 a076baae5fab7ef7 · raw

final URL: manual:file

1
2# For all robots
3User-agent: *
4
5# Block access to specific groups of pages
6Disallow: */cart/
7Disallow: */checkout/
8Disallow: */my-account/
9Disallow: */school/
10
11
12# Block common duplicate/low-value parameterized URLs
13Disallow: /*?sessionid=
14Disallow: /*?ref=
15Disallow: /*?utm_
16
17# Allow search crawlers to discover the sitemap
18Sitemap: https://www.wiley.com/sitemap.xml
19Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.products-sitemap1.xml
20Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.products-sitemap2.xml
21Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.products-sitemap3.xml
22Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.products-sitemap4.xml
23Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.products-sitemap5.xml
24Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.products-sitemap6.xml
25Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.categories-sitemap1.xml
26Sitemap: https://www.wiley.com/content/wiley-com/apac/in/en.sitemap.content.xml
27Sitemap: https://www.wiley.com/content/wiley-com/apac/kr/en.sitemap.content.xml
28Sitemap: https://www.wiley.com/content/wiley-com/apac/hk/en.sitemap.content.xml
29Sitemap: https://www.wiley.com/content/wiley-com/apac/sg/en.sitemap.content.xml
30Sitemap: https://www.wiley.com/content/wiley-com/apac/nz/en.sitemap.content.xml
31Sitemap: https://www.wiley.com/content/wiley-com/apac/cn/en.sitemap.content.xml
32Sitemap: https://www.wiley.com/content/wiley-com/apac/jp/en.sitemap.content.xml
33Sitemap: https://www.wiley.com/content/wiley-com/apac/au/en.sitemap.content.xml
34Sitemap: https://www.wiley.com/content/wiley-com/na/us/en.sitemap.content.xml
35Sitemap: https://www.wiley.com/content/wiley-com/ca/us/en.sitemap.content.xml
36Sitemap: https://www.wiley.com/content/wiley-com/latam/br/en.sitemap.content.xml
37Sitemap: https://www.wiley.com/content/wiley-com/latam/mx/en.sitemap.content.xml
38Sitemap: https://www.wiley.com/content/wiley-com/emea/ae/en.sitemap.content.xml
39Sitemap: https://www.wiley.com/content/wiley-com/emea/nl/en.sitemap.content.xml
40Sitemap: https://www.wiley.com/content/wiley-com/emea/no/en.sitemap.content.xml
41Sitemap: https://www.wiley.com/content/wiley-com/emea/dk/en.sitemap.content.xml
42Sitemap: https://www.wiley.com/content/wiley-com/emea/es/en.sitemap.content.xml
43Sitemap: https://www.wiley.com/content/wiley-com/emea/it/en.sitemap.content.xml
44Sitemap: https://www.wiley.com/content/wiley-com/emea/se/en.sitemap.content.xml
45Sitemap: https://www.wiley.com/content/wiley-com/emea/be/en.sitemap.content.xml
46Sitemap: https://www.wiley.com/content/wiley-com/emea/fr/en.sitemap.content.xml
47Sitemap: https://www.wiley.com/content/wiley-com/emea/ie/en.sitemap.content.xml
48Sitemap: https://www.wiley.com/content/wiley-com/emea/gb/en.sitemap.content.xml
49Sitemap: https://www.wiley.com/content/wiley-com/emea/de/en.sitemap.content.xml
50
51
52# Block unwanted crawlers
53User-agent: CazoodleBot
54Disallow: /
55
56User-agent: MJ12bot
57Disallow: /
58
59User-agent: dotbot*
60Disallow: /
61
62User-agent: Gigabot
63Disallow: /
64
65User-agent: trendkite-akashic-crawler
66Disallow: /
67
68User-agent: AhrefsBot
69Disallow: /
70
71User-agent: MauiBot
72Disallow: /
73
74User-agent: Barkrowler
75Disallow: /