NeuralCrawl

Evonik / robots.txt snapshot

← back to evonik.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 1201 bytes · sha256 2ff9f4fc7b1f1150 · raw

final URL: https://www.evonik.com/robots.txt

1# Start Group: crawler / LLMO
2User-Agent: Bingbot
3User-Agent: GPTBot
4User-Agent: ClaudeBot
5User-Agent: Claude-User
6User-Agent: Claude-SearchBot
7User-Agent: CCBot
8User-Agent: Google-Extended
9User-Agent: Applebot-Extended
10User-Agent: Facebookbot
11User-Agent: Meta-ExternalAgent
12User-Agent: Meta-ExternalFetcher
13User-Agent: diffbot
14User-Agent: PerplexityBot
15User-Agent: TikTokSpider
16User-Agent: Amazonbot
17User-Agent: OAI-SearchBot
18User-Agent: Google-CloudVertexBot
19User-Agent: DuckAssistBot
20User-Agent: GoogleAgent-Mariner
21User-Agent: Gemini-Deep-Research
22User-Agent: Google-NotebookLM
23User-Agent: Google-Agent
24User-Agent: GoogleAgent-URLContext
25User-Agent: Google-Firebase
26User-Agent: MistralAI-User
27User-Agent: Spacecat/1.0
28Disallow: /en/my-evonik/
29Disallow: /en/incident/
30Disallow: /de/incident/
31Disallow: /en/error/
32Allow: /
33# End Group: crawler / LLMO
34# Start Group: known Spam-Bots
35User-agent: MJ12bot
36Disallow: /
37# End Group: known Spam-Bots
38# Start Group: all others
39User-agent: *
40Disallow: /en/my-evonik/
41Disallow: /en/incident/
42Disallow: /de/incident/
43Disallow: /en/error/
44Allow: /
45# End Group: all
46
47Sitemap: https://www.evonik.com/en.sitemap.xml
48Sitemap: https://www.evonik.com/de.sitemap.xml