Zscaler / robots.txt

← back to zscaler.com · change detected 11h ago (2026-08-04T12:42:22Z) · modified +24 −1

What changed

🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
🤖 AIAdded rules for AI crawler cohere-ai (Cohere)
🤖 AIAdded rules for AI crawler Google-Extended (Google)
🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
Added Allow: / for *
Removed Allow: /*/blogs?topic=* for *
Removed Allow: /*/blogs?type=* for *
Removed Allow: /*/careers/search?office=* for *
Removed Allow: /*/resources/webinars?topic=* for *
Removed Allow: /*/resources/webinars?type=* for *
Removed Allow: /*/resources?topic=* for *
Removed Allow: /*/resources?type=* for *
Removed Allow: /*/zpedia?type=* for *
Removed Allow: /_next/image?url=* for *
Removed Allow: /blogs?topic=* for *
Removed Allow: /blogs?type=* for *
Removed Allow: /careers/search?office=* for *
Removed Allow: /core/*.css$ for *
Removed Allow: /core/*.css? for *
Removed Allow: /core/*.gif for *
Removed Allow: /core/*.jpeg for *
Removed Allow: /core/*.jpg for *
Removed Allow: /core/*.js$ for *
Removed Allow: /core/*.js? for *
Removed Allow: /core/*.png for *
Removed Allow: /core/*.svg for *
Removed Allow: /profiles/*.css$ for *
Removed Allow: /profiles/*.css? for *
Removed Allow: /profiles/*.gif for *
Removed Allow: /profiles/*.jpeg for *
Removed Allow: /profiles/*.jpg for *
Removed Allow: /profiles/*.js$ for *
Removed Allow: /profiles/*.js? for *
Removed Allow: /profiles/*.png for *
Removed Allow: /profiles/*.svg for *
Removed Allow: /resources/webinars?topic=* for *
Removed Allow: /resources/webinars?type=* for *
Removed Allow: /resources?topic=* for *
Removed Allow: /resources?type=* for *
Removed Allow: /themes/*.css$ for *
Removed Allow: /themes/*.css? for *
Removed Allow: /themes/*.gif for *
Removed Allow: /themes/*.jpeg for *
Removed Allow: /themes/*.jpg for *
... and 63 more change(s)

Diff old (2026-06-20T01:10:31Z) → new (2026-08-04T12:42:22Z)

		@@ -1,6 +1,29 @@
1	1
2	2	User-agent: *
3		- Crawl-delay: 10
	3	+ Allow: /
	4	+ # ======================================================================
	5	+ # AI/LLM Friendly Content Locations
	6	+ # See https://llmstxt.org for the llms.txt specification
	7	+ #
	8	+ # - /llms.txt - Curated overview for AI/LLMs (markdown)
	9	+ # - /llms-full.txt - Full expanded content for larger context windows
	10	+ # - /answers.json - Structured QA/answer feed for AI engines
	11	+ # ======================================================================
	12	+
	13	+ # Allow AI search engines and user-triggered LLM crawlers
	14	+ User-agent: OAI-SearchBot
	15	+ User-agent: ChatGPT-User
	16	+ User-agent: PerplexityBot
	17	+ User-agent: Claude-SearchBot
	18	+ User-agent: Claude-User
	19	+ User-agent: ClaudeBot
	20	+ User-agent: GPTBot
	21	+ User-agent: Google-Extended
	22	+ User-agent: Cohere-ai
	23	+ User-agent: Applebot-Extended
	24	+ Allow: /
	25	+
	26	+
4	27
5	28	Allow: //resources?topic=
6	29	Allow: //blogs?type=