Kimberly-Clark / robots.txt
← back to kimberly-clark.com · change detected 1mo ago (2026-09-29T06:46:46Z) · modified +2 −109
What changed
- Removed all rules for user-agent ‘AndiBot’
- 🤖 AIUnblocked anthropic-ai (Anthropic) - rules removed
- Removed all rules for user-agent ‘Applebot’
- 🤖 AIUnblocked Applebot-Extended (Apple) - rules removed
- Removed all rules for user-agent ‘Bingbot’
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- 🤖 AIUnblocked CCBot (Common Crawl) - rules removed
- 🤖 AIRemoved rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIRemoved rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIRemoved rules for AI crawler Claude-User (Anthropic)
- 🤖 AIUnblocked ClaudeBot (Anthropic) - rules removed
- 🤖 AIUnblocked cohere-ai (Cohere) - rules removed
- 🤖 AIRemoved rules for AI crawler Diffbot (Diffbot)
- 🤖 AIRemoved rules for AI crawler DuckAssistBot (DuckDuckGo)
- 🤖 AIRemoved rules for AI crawler FacebookBot (Meta)
- Removed all rules for user-agent ‘Gemini-Deep-Research’
- Removed all rules for user-agent ‘Google-CloudVertexBot’
- 🤖 AIUnblocked Google-Extended (Google) - rules removed
- Removed all rules for user-agent ‘Google-InspectionTool’
- Removed all rules for user-agent ‘Googlebot’
- 🤖 AIUnblocked GPTBot (OpenAI) - rules removed
- Removed all rules for user-agent ‘LinkedInBot’
- 🤖 AIUnblocked meta-externalagent (Meta) - rules removed
- 🤖 AIRemoved rules for AI crawler meta-externalfetcher (Meta)
- 🤖 AIRemoved rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIRemoved rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIRemoved rules for AI crawler PerplexityBot (Perplexity)
- Removed all rules for user-agent ‘PhindBot’
- Removed all rules for user-agent ‘QAI-SearchBot’
- Removed all rules for user-agent ‘Storebot-Google’
- Blocked user-agent ‘*’ entirely
- Removed Allow: *.css for *
- Removed Allow: *.jpg for *
- Removed Allow: *.js for *
- Removed Allow: *.pdf for *
- Removed Allow: *.png for *
- Removed Allow: /-/*.svg for *
- Removed Allow: /js/*.js for *
- Removed Allow: /scripts/*.js for *
- Removed Disallow: *& for *
- Removed Disallow: *.asp for *
- Removed Disallow: *.swf for *
- Removed Disallow: */old for *
- Removed Disallow: *?from=articles for *
- Removed Disallow: *?q for *
- Removed Disallow: *?type for *
- Removed Disallow: *?url for *
- Removed Disallow: *?version for *
- Removed Disallow: *clid for *
- Removed Disallow: *email-protection for *
- ... and 37 more change(s)
Diff old (2026-06-29T11:44:49Z) → new (2026-09-29T06:46:46Z)
| @@ -1,109 +1,2 @@ | ||
| 1 | - # AI Search & Answer Engines (no AI training; search, citations, user‑initiated fetch only) | |
| 2 | - User-agent: OAI-SearchBot | |
| 3 | - User-agent: ChatGPT-User | |
| 4 | - User-agent: Claude-SearchBot | |
| 5 | - User-agent: Claude-User | |
| 6 | - User-agent: PerplexityBot | |
| 7 | - User-agent: Perplexity-User | |
| 8 | - User-agent: DuckAssistBot | |
| 9 | - User-agent: Applebot | |
| 10 | - User-agent: Meta-ExternalFetcher | |
| 11 | - User-agent: Google-CloudVertexBot | |
| 12 | - User-agent: Diffbot | |
| 13 | - User-agent: Gemini-Deep-Research | |
| 14 | - User-agent: FacebookBot | |
| 15 | - User-agent: LinkedInBot | |
| 16 | - User-agent: Storebot-Google | |
| 17 | - User-agent: Google-InspectionTool | |
| 18 | - Allow: / | |
| 19 | - | |
| 20 | - # Third‑party AI / SERP tooling (explicitly allowed, non‑training) | |
| 21 | - User-agent: AndiBot | |
| 22 | - User-agent: PhindBot | |
| 23 | - User-agent: QAI-SearchBot | |
| 24 | - Allow: / | |
| 25 | - | |
| 26 | - # Traditional search engines | |
| 27 | - User-agent: Googlebot | |
| 28 | - User-agent: Bingbot | |
| 29 | - Allow: / | |
| 30 | - | |
| 31 | - # Disallow AI training & dataset collection | |
| 32 | - User-agent: GPTBot | |
| 33 | - User-agent: ClaudeBot | |
| 34 | - User-agent: anthropic-ai | |
| 35 | - User-agent: Google-Extended | |
| 36 | - User-agent: Applebot-Extended | |
| 37 | - User-agent: Meta-ExternalAgent | |
| 38 | - User-agent: CCBot | |
| 39 | - User-agent: Bytespider | |
| 40 | - User-agent: cohere-ai | |
| 41 | - Disallow: / | |
| 42 | - | |
| 43 | - # Disallow sensitive or non-public areas for all crawlers | |
| 44 | - User-agent: * | |
| 45 | - Disallow: /Config/ | |
| 46 | - Disallow: /bin/ | |
| 47 | - Disallow: /config/ | |
| 48 | - Disallow: /data/ | |
| 49 | - Disallow: /install/ | |
| 50 | - Disallow: /masterpages/ | |
| 51 | - Disallow: /python/ | |
| 52 | - Disallow: /scripts/ | |
| 53 | - Disallow: /umbraco/ | |
| 54 | - Disallow: /xslt/ | |
| 55 | - Disallow: /cdn-cgi/ | |
| 56 | - | |
| 57 | - # Query & pattern-based exclusions | |
| 58 | - Disallow: *user | |
| 59 | - Disallow: *& | |
| 60 | - Disallow: *clid | |
| 61 | - Disallow: *openstat | |
| 62 | - Disallow: *utm | |
| 63 | - Disallow: *search | |
| 64 | - Disallow: *result | |
| 65 | - Disallow: *email-protection | |
| 66 | - Disallow: *?version | |
| 67 | - Disallow: *?url | |
| 68 | - Disallow: *?type | |
| 69 | - Disallow: *?q | |
| 70 | - Disallow: *?from=articles | |
| 71 | - Disallow: *external-test | |
| 72 | - Disallow: *mindbox-click-id | |
| 73 | - Disallow: */old | |
| 74 | - | |
| 75 | - # Functional paths | |
| 76 | - Disallow: /search | |
| 77 | - Disallow: /articlesearch | |
| 78 | - Disallow: /account | |
| 79 | - Disallow: /tags | |
| 80 | - Disallow: /categories | |
| 81 | - Disallow: /subscriptions | |
| 82 | - Disallow: /recovery | |
| 83 | - Disallow: /Error | |
| 84 | - Disallow: /Home | |
| 85 | - | |
| 86 | - # File types | |
| 87 | - Disallow: *.asp | |
| 88 | - Disallow: *.swf | |
| 89 | - | |
| 90 | - # Explicitly allowed assets | |
| 91 | - Allow: /-/*.svg | |
| 92 | - Allow: /scripts/*.js | |
| 93 | - Allow: /js/*.js | |
| 94 | - Allow: *.css | |
| 95 | - Allow: *.js | |
| 96 | - Allow: *.png | |
| 97 | - Allow: *.jpg | |
| 98 | - Allow: *.pdf | |
| 99 | - | |
| 100 | - Sitemap: https://www.kimberly-clark.com/en-vn/sitemap.xml | |
| 101 | - Sitemap: https://www.kimberly-clark.com/en-us/sitemap.xml | |
| 102 | - Sitemap: https://www.kimberly-clark.com/es-us/sitemap.xml | |
| 103 | - Sitemap: https://www.kimberly-clark.com/en-ca/sitemap.xml | |
| 104 | - Sitemap: https://www.kimberly-clark.com/fr-ca/sitemap.xml | |
| 105 | - Sitemap: https://www.kimberly-clark.com/pt-br/sitemap.xml | |
| 106 | - Sitemap: https://www.kimberly-clark.com/en-au/sitemap.xml | |
| 107 | - Sitemap: https://www.kimberly-clark.com/ru-ru/sitemap.xml | |
| 108 | - Sitemap: https://www.kimberly-clark.com/uk-ua/sitemap.xml | |
| 109 | - Sitemap: https://www.kimberly-clark.com/zh-tw/sitemap.xml | |
| 1 | + User-agent: * | |
| 2 | + Disallow: / |