Mondelez International / robots.txt

← back to mondelezinternational.com · change detected 30d ago (2026-10-08T10:00:34Z) · modified +55 −0

What changed

Added rules for user-agent ‘bingbot’
🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
🤖 AIAdded rules for AI crawler Google-Extended (Google)
Added rules for user-agent ‘Googlebot’
🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)

Diff old (2026-06-20T01:10:30Z) → new (2026-10-08T10:00:34Z)

		@@ -1,4 +1,59 @@
1	1	User-agent: *
	2	+ Allow: /
	3	+ Disallow: /assets/
	4	+ Disallow: /*.pdf$
	5	+
	6	+ User-agent: OAI-SearchBot
	7	+ Allow: /
	8	+ Disallow: /assets/
	9	+ Disallow: /*.pdf$
	10	+
	11	+ User-agent: ChatGPT-User
	12	+ Allow: /
	13	+ Disallow: /assets/
	14	+ Disallow: /*.pdf$
	15	+
	16	+ User-agent: GPTBot
	17	+ Allow: /
	18	+ Disallow: /assets/
	19	+ Disallow: /*.pdf$
	20	+
	21	+ User-agent: Claude-SearchBot
	22	+ Allow: /
	23	+ Disallow: /assets/
	24	+ Disallow: /*.pdf$
	25	+
	26	+ User-agent: Claude-User
	27	+ Allow: /
	28	+ Disallow: /assets/
	29	+ Disallow: /*.pdf$
	30	+
	31	+ User-agent: ClaudeBot
	32	+ Allow: /
	33	+ Disallow: /assets/
	34	+ Disallow: /*.pdf$
	35	+
	36	+ User-agent: PerplexityBot
	37	+ Allow: /
	38	+ Disallow: /assets/
	39	+ Disallow: /*.pdf$
	40	+
	41	+ User-agent: Perplexity-User
	42	+ Allow: /
	43	+ Disallow: /assets/
	44	+ Disallow: /*.pdf$
	45	+
	46	+ User-agent: Google-Extended
	47	+ Allow: /
	48	+ Disallow: /assets/
	49	+ Disallow: /*.pdf$
	50	+
	51	+ User-agent: Googlebot
	52	+ Allow: /
	53	+ Disallow: /assets/
	54	+ Disallow: /*.pdf$
	55	+
	56	+ User-agent: bingbot
2	57	Allow: /
3	58	Disallow: /assets/
4	59	Disallow: /*.pdf$