Celanese / robots.txt
← back to celanese.com · change detected 2mo ago (2026-09-06T23:40:27Z) · modified +7 −0
What changed
- Added rules for user-agent ‘*’
Diff old (2026-06-20T01:10:30Z) → new (2026-09-06T23:40:27Z)
| @@ -1,3 +1,10 @@ | ||
| 1 | + Disallow: /sitecore_files/ | |
| 2 | + Disallow: /sitecore modules/ | |
| 3 | + Disallow: /App_Browsers/ | |
| 4 | + Disallow: /App_config/ | |
| 5 | + Disallow: /App_Data/ | |
| 6 | + Disallow: /temp/ | |
| 7 | + Disallow: /xsl/ | |
| 1 | 8 | User-agent: Screaming Frog SEO Spider |
| 2 | 9 | Allow: / |
| 3 | 10 |