Disallow: /sitecore_files/ Disallow: /sitecore modules/ Disallow: /App_Browsers/ Disallow: /App_config/ Disallow: /App_Data/ Disallow: /temp/ Disallow: /xsl/ User-agent: Screaming Frog SEO Spider Allow: / User-agent: rogerbot Allow: / Sitemap: https://www.celanese.com/sitemap