Evonik / robots.txt

What changed

• Response is a valid robots.txt again (was an HTML page)
• Added rules for user-agent ‘*’
• 🤖 AIAdded rules for AI crawler Amazonbot (Amazon)
• 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
• Added rules for user-agent ‘Bingbot’
• 🤖 AIAdded rules for AI crawler CCBot (Common Crawl)
• 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
• 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
• 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
• 🤖 AIAdded rules for AI crawler Diffbot (Diffbot)
• 🤖 AIAdded rules for AI crawler DuckAssistBot (DuckDuckGo)
• 🤖 AIAdded rules for AI crawler FacebookBot (Meta)
• Added rules for user-agent ‘Gemini-Deep-Research’
• Added rules for user-agent ‘Google-Agent’
• Added rules for user-agent ‘Google-CloudVertexBot’
• 🤖 AIAdded rules for AI crawler Google-Extended (Google)
• Added rules for user-agent ‘Google-Firebase’
• Added rules for user-agent ‘Google-NotebookLM’
• Added rules for user-agent ‘GoogleAgent-Mariner’
• Added rules for user-agent ‘GoogleAgent-URLContext’
• 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
• 🤖 AIAdded rules for AI crawler meta-externalagent (Meta)
• 🤖 AIAdded rules for AI crawler meta-externalfetcher (Meta)
• 🤖 AIAdded rules for AI crawler MistralAI-User (Mistral AI)
• Blocked user-agent ‘MJ12bot’ entirely
• 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
• 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
• Added rules for user-agent ‘Spacecat/1.0’
• Added rules for user-agent ‘TikTokSpider’
• Added sitemap https://www.evonik.com/de.sitemap.xml
• Added sitemap https://www.evonik.com/en.sitemap.xml

Diff (no previous version) → (file removed)

No line-level differences (identical or both empty).