Kimberly-Clark / robots.txt

← back to kimberly-clark.com · change detected 1mo ago (2026-09-29T12:52:06Z) · modified +109 −2

What changed

Added rules for user-agent ‘AndiBot’
🤖 AIBlocked anthropic-ai (Anthropic) entirely
Added rules for user-agent ‘Applebot’
🤖 AIBlocked Applebot-Extended (Apple) entirely
Added rules for user-agent ‘Bingbot’
🤖 AIBlocked Bytespider (ByteDance) entirely
🤖 AIBlocked CCBot (Common Crawl) entirely
🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
🤖 AIBlocked ClaudeBot (Anthropic) entirely
🤖 AIBlocked cohere-ai (Cohere) entirely
🤖 AIAdded rules for AI crawler Diffbot (Diffbot)
🤖 AIAdded rules for AI crawler DuckAssistBot (DuckDuckGo)
🤖 AIAdded rules for AI crawler FacebookBot (Meta)
Added rules for user-agent ‘Gemini-Deep-Research’
Added rules for user-agent ‘Google-CloudVertexBot’
🤖 AIBlocked Google-Extended (Google) entirely
Added rules for user-agent ‘Google-InspectionTool’
Added rules for user-agent ‘Googlebot’
🤖 AIBlocked GPTBot (OpenAI) entirely
Added rules for user-agent ‘LinkedInBot’
🤖 AIBlocked meta-externalagent (Meta) entirely
🤖 AIAdded rules for AI crawler meta-externalfetcher (Meta)
🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
Added rules for user-agent ‘PhindBot’
Added rules for user-agent ‘QAI-SearchBot’
Added rules for user-agent ‘Storebot-Google’
Unblocked user-agent ‘*’
Added Allow: *.css for *
Added Allow: *.jpg for *
Added Allow: *.js for *
Added Allow: *.pdf for *
Added Allow: *.png for *
Added Allow: /-/*.svg for *
Added Allow: /js/*.js for *
Added Allow: /scripts/*.js for *
Added Disallow: *& for *
Added Disallow: *.asp for *
Added Disallow: *.swf for *
Added Disallow: */old for *
Added Disallow: *?from=articles for *
Added Disallow: *?q for *
Added Disallow: *?type for *
Added Disallow: *?url for *
Added Disallow: *?version for *
Added Disallow: *clid for *
Added Disallow: *email-protection for *
... and 37 more change(s)

Diff old (2026-09-29T06:46:46Z) → new (2026-09-29T12:52:06Z)

		@@ -1,2 +1,109 @@
1		- User-agent: *
2		- Disallow: /
	1	+ # AI Search & Answer Engines (no AI training; search, citations, user‑initiated fetch only)
	2	+ User-agent: OAI-SearchBot
	3	+ User-agent: ChatGPT-User
	4	+ User-agent: Claude-SearchBot
	5	+ User-agent: Claude-User
	6	+ User-agent: PerplexityBot
	7	+ User-agent: Perplexity-User
	8	+ User-agent: DuckAssistBot
	9	+ User-agent: Applebot
	10	+ User-agent: Meta-ExternalFetcher
	11	+ User-agent: Google-CloudVertexBot
	12	+ User-agent: Diffbot
	13	+ User-agent: Gemini-Deep-Research
	14	+ User-agent: FacebookBot
	15	+ User-agent: LinkedInBot
	16	+ User-agent: Storebot-Google
	17	+ User-agent: Google-InspectionTool
	18	+ Allow: /
	19	+
	20	+ # Third‑party AI / SERP tooling (explicitly allowed, non‑training)
	21	+ User-agent: AndiBot
	22	+ User-agent: PhindBot
	23	+ User-agent: QAI-SearchBot
	24	+ Allow: /
	25	+
	26	+ # Traditional search engines
	27	+ User-agent: Googlebot
	28	+ User-agent: Bingbot
	29	+ Allow: /
	30	+
	31	+ # Disallow AI training & dataset collection
	32	+ User-agent: GPTBot
	33	+ User-agent: ClaudeBot
	34	+ User-agent: anthropic-ai
	35	+ User-agent: Google-Extended
	36	+ User-agent: Applebot-Extended
	37	+ User-agent: Meta-ExternalAgent
	38	+ User-agent: CCBot
	39	+ User-agent: Bytespider
	40	+ User-agent: cohere-ai
	41	+ Disallow: /
	42	+
	43	+ # Disallow sensitive or non-public areas for all crawlers
	44	+ User-agent: *
	45	+ Disallow: /Config/
	46	+ Disallow: /bin/
	47	+ Disallow: /config/
	48	+ Disallow: /data/
	49	+ Disallow: /install/
	50	+ Disallow: /masterpages/
	51	+ Disallow: /python/
	52	+ Disallow: /scripts/
	53	+ Disallow: /umbraco/
	54	+ Disallow: /xslt/
	55	+ Disallow: /cdn-cgi/
	56	+
	57	+ # Query & pattern-based exclusions
	58	+ Disallow: *user
	59	+ Disallow: *&
	60	+ Disallow: *clid
	61	+ Disallow: *openstat
	62	+ Disallow: *utm
	63	+ Disallow: *search
	64	+ Disallow: *result
	65	+ Disallow: *email-protection
	66	+ Disallow: *?version
	67	+ Disallow: *?url
	68	+ Disallow: *?type
	69	+ Disallow: *?q
	70	+ Disallow: *?from=articles
	71	+ Disallow: *external-test
	72	+ Disallow: *mindbox-click-id
	73	+ Disallow: */old
	74	+
	75	+ # Functional paths
	76	+ Disallow: /search
	77	+ Disallow: /articlesearch
	78	+ Disallow: /account
	79	+ Disallow: /tags
	80	+ Disallow: /categories
	81	+ Disallow: /subscriptions
	82	+ Disallow: /recovery
	83	+ Disallow: /Error
	84	+ Disallow: /Home
	85	+
	86	+ # File types
	87	+ Disallow: *.asp
	88	+ Disallow: *.swf
	89	+
	90	+ # Explicitly allowed assets
	91	+ Allow: /-/*.svg
	92	+ Allow: /scripts/*.js
	93	+ Allow: /js/*.js
	94	+ Allow: *.css
	95	+ Allow: *.js
	96	+ Allow: *.png
	97	+ Allow: *.jpg
	98	+ Allow: *.pdf
	99	+
	100	+ Sitemap: https://www.kimberly-clark.com/en-vn/sitemap.xml
	101	+ Sitemap: https://www.kimberly-clark.com/en-us/sitemap.xml
	102	+ Sitemap: https://www.kimberly-clark.com/es-us/sitemap.xml
	103	+ Sitemap: https://www.kimberly-clark.com/en-ca/sitemap.xml
	104	+ Sitemap: https://www.kimberly-clark.com/fr-ca/sitemap.xml
	105	+ Sitemap: https://www.kimberly-clark.com/pt-br/sitemap.xml
	106	+ Sitemap: https://www.kimberly-clark.com/en-au/sitemap.xml
	107	+ Sitemap: https://www.kimberly-clark.com/ru-ru/sitemap.xml
	108	+ Sitemap: https://www.kimberly-clark.com/uk-ua/sitemap.xml
	109	+ Sitemap: https://www.kimberly-clark.com/zh-tw/sitemap.xml