Kimberly-Clark / robots.txt snapshot
← back to kimberly-clark.com · fetched 2026-06-29T11:44:49Z (1mo ago) · HTTP 200 · 2643 bytes · sha256 32993400e74b784e · raw
final URL: https://www.kimberly-clark.com/robots.txt
| 1 | # AI Search & Answer Engines (no AI training; search, citations, user‑initiated fetch only) |
| 2 | User-agent: OAI-SearchBot |
| 3 | User-agent: ChatGPT-User |
| 4 | User-agent: Claude-SearchBot |
| 5 | User-agent: Claude-User |
| 6 | User-agent: PerplexityBot |
| 7 | User-agent: Perplexity-User |
| 8 | User-agent: DuckAssistBot |
| 9 | User-agent: Applebot |
| 10 | User-agent: Meta-ExternalFetcher |
| 11 | User-agent: Google-CloudVertexBot |
| 12 | User-agent: Diffbot |
| 13 | User-agent: Gemini-Deep-Research |
| 14 | User-agent: FacebookBot |
| 15 | User-agent: LinkedInBot |
| 16 | User-agent: Storebot-Google |
| 17 | User-agent: Google-InspectionTool |
| 18 | Allow: / |
| 19 | |
| 20 | # Third‑party AI / SERP tooling (explicitly allowed, non‑training) |
| 21 | User-agent: AndiBot |
| 22 | User-agent: PhindBot |
| 23 | User-agent: QAI-SearchBot |
| 24 | Allow: / |
| 25 | |
| 26 | # Traditional search engines |
| 27 | User-agent: Googlebot |
| 28 | User-agent: Bingbot |
| 29 | Allow: / |
| 30 | |
| 31 | # Disallow AI training & dataset collection |
| 32 | User-agent: GPTBot |
| 33 | User-agent: ClaudeBot |
| 34 | User-agent: anthropic-ai |
| 35 | User-agent: Google-Extended |
| 36 | User-agent: Applebot-Extended |
| 37 | User-agent: Meta-ExternalAgent |
| 38 | User-agent: CCBot |
| 39 | User-agent: Bytespider |
| 40 | User-agent: cohere-ai |
| 41 | Disallow: / |
| 42 | |
| 43 | # Disallow sensitive or non-public areas for all crawlers |
| 44 | User-agent: * |
| 45 | Disallow: /Config/ |
| 46 | Disallow: /bin/ |
| 47 | Disallow: /config/ |
| 48 | Disallow: /data/ |
| 49 | Disallow: /install/ |
| 50 | Disallow: /masterpages/ |
| 51 | Disallow: /python/ |
| 52 | Disallow: /scripts/ |
| 53 | Disallow: /umbraco/ |
| 54 | Disallow: /xslt/ |
| 55 | Disallow: /cdn-cgi/ |
| 56 | |
| 57 | # Query & pattern-based exclusions |
| 58 | Disallow: *user |
| 59 | Disallow: *& |
| 60 | Disallow: *clid |
| 61 | Disallow: *openstat |
| 62 | Disallow: *utm |
| 63 | Disallow: *search |
| 64 | Disallow: *result |
| 65 | Disallow: *email-protection |
| 66 | Disallow: *?version |
| 67 | Disallow: *?url |
| 68 | Disallow: *?type |
| 69 | Disallow: *?q |
| 70 | Disallow: *?from=articles |
| 71 | Disallow: *external-test |
| 72 | Disallow: *mindbox-click-id |
| 73 | Disallow: */old |
| 74 | |
| 75 | # Functional paths |
| 76 | Disallow: /search |
| 77 | Disallow: /articlesearch |
| 78 | Disallow: /account |
| 79 | Disallow: /tags |
| 80 | Disallow: /categories |
| 81 | Disallow: /subscriptions |
| 82 | Disallow: /recovery |
| 83 | Disallow: /Error |
| 84 | Disallow: /Home |
| 85 | |
| 86 | # File types |
| 87 | Disallow: *.asp |
| 88 | Disallow: *.swf |
| 89 | |
| 90 | # Explicitly allowed assets |
| 91 | Allow: /-/*.svg |
| 92 | Allow: /scripts/*.js |
| 93 | Allow: /js/*.js |
| 94 | Allow: *.css |
| 95 | Allow: *.js |
| 96 | Allow: *.png |
| 97 | Allow: *.jpg |
| 98 | Allow: *.pdf |
| 99 | |
| 100 | Sitemap: https://www.kimberly-clark.com/en-vn/sitemap.xml |
| 101 | Sitemap: https://www.kimberly-clark.com/en-us/sitemap.xml |
| 102 | Sitemap: https://www.kimberly-clark.com/es-us/sitemap.xml |
| 103 | Sitemap: https://www.kimberly-clark.com/en-ca/sitemap.xml |
| 104 | Sitemap: https://www.kimberly-clark.com/fr-ca/sitemap.xml |
| 105 | Sitemap: https://www.kimberly-clark.com/pt-br/sitemap.xml |
| 106 | Sitemap: https://www.kimberly-clark.com/en-au/sitemap.xml |
| 107 | Sitemap: https://www.kimberly-clark.com/ru-ru/sitemap.xml |
| 108 | Sitemap: https://www.kimberly-clark.com/uk-ua/sitemap.xml |
| 109 | Sitemap: https://www.kimberly-clark.com/zh-tw/sitemap.xml |