NeuralCrawl

CSL / robots.txt

← back to csl.com · change detected 2mo ago (2026-08-31T21:34:42Z) · modified +0 −11

What changed

Diff old (2026-06-20T01:10:31Z) → new (2026-08-31T21:34:42Z)

@@ -1,13 +1,2 @@
11  User-agent: *
22  Sitemap: https://www.csl.com/sitemap.xml
3-
4- Disallow: /Sitecore
5- Disallow: /sitecore_files/
6- Disallow: /sitecore modules/
7- Disallow: /App_Browsers/
8- Disallow: /App_config/
9- Disallow: /App_Data/
10- Disallow: /temp/
11- Disallow: /upload/
12- Disallow: /xsl/
13-