NeuralCrawl

๐Ÿ‡ฉ๐Ÿ‡ช Continental

continental.com · European companies · rank #91 · Motor Vehicles · live robots.txt ↗

AI crawler access (latest snapshot, 13h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 1702 bytes · sha256 023fb6a0a0cc · raw

User-Agent: *
Allow: / 
# Only allow URLs generated with frontend routing
Disallow: /*?id=*
Disallow: /*&id=*
# L=0 is the default language
Disallow: /*?L=0*
Disallow: /*&L=0*

# Disallow API PageTypes for jobportal
Disallow: */api/*/pagetype-jobs/$

# Disallow folders
# Disallow: /typo3
# Sitemap: International
sitemap: https://www.continental.com/en/xmlsitemap
sitemap: https://www.continental.com/de/xmlsitemap
# Sitemap: America
sitemap: https://www.continental.com/en-us/xmlsitemap
sitemap: https://www.continental.com/es-mx/xmlsitemap
# Sitemap: Asia
sitemap: https://www.continental.com/zh-cn/xmlsitemap
sitemap: https://www.continental.com/ja-jp/xmlsitemap
sitemap: https://www.continental.com/ko-kr/xmlsitemap
#Sitemap: Europe
sitemap: https://www.continental.com/cs-cz/xmlsitemap
sitemap: https://www.continental.com/fr-fr/xmlsitemap
sitemap: https://www.continental.com/hu-hu/xmlsitemap
sitemap: https://www.continental.com/ro-ro/xmlsitemap
#Sitemap: Jobportal
sitemap: https://jobs.continental.com/en/xmlsitemap
sitemap: https://jobs.continental.com/de/xmlsitemap
sitemap: https://jobs.continental.com/cz/xmlsitemap
sitemap: https://jobs.continental.com/zh/xmlsitemap
sitemap: https://jobs.continental.com/fr/xmlsitemap
sitemap: https://jobs.continental.com/es/xmlsitemap
sitemap: https://jobs.continental.com/ja/xmlsitemap
sitemap: https://jobs.continental.com/ko/xmlsitemap
sitemap: https://jobs.continental.com/lt/xmlsitemap
sitemap: https://jobs.continental.com/hu/xmlsitemap
sitemap: https://jobs.continental.com/ro/xmlsitemap
sitemap: https://jobs.continental.com/pt/xmlsitemap
sitemap: https://jobs.continental.com/sk/xmlsitemap
sitemap: https://jobs.continental.com/sr/xmlsitemap

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived