NeuralCrawl

๐Ÿ‡ฉ๐Ÿ‡ช Thyssenkrupp

thyssenkrupp.com · European companies · rank #55 · Industrials · live robots.txt ↗

AI crawler access (latest snapshot, 13h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 300 bytes · sha256 3c60bee00d6d · raw

User-agent: * 
Disallow: /de/home-2604
Disallow: /_api
Disallow: /en/newsroom/press-releases/press-release-
Disallow: /de/newsroom/pressemeldungen/press-release-
Disallow: /hu
Disallow: *UCPthyssenkruppAG*
Disallow: /_media/
Disallow: /assets.files/
Sitemap: https://www.thyssenkrupp.com/sitemap.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived