NeuralCrawl

πŸ‡ΊπŸ‡Έ Illumio

illumio.com · Cybersecurity · rank #45 · Cybersecurity · live robots.txt ↗

AI crawler access (latest snapshot, 5h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 1028 bytes · sha256 ddfaaa264dfb · raw

User-agent: *
Disallow: /search?query=*
Disallow: /*?
Disallow: /*/news/press-releases/*
Disallow: /*/resources-industries/*
Disallow: /*/resources-products/*
Disallow: /*/resources-topics/*
Disallow: /*/resource-center/solution-brief/*
Disallow: /*/resource-center/research-report/*
Disallow: /bugs/
Disallow: /resource-center/a-simple-guide-to-stopping-the-spread-of-breaches
Disallow: /node/
Disallow: /sites/
Disallow: /ja/node/
Disallow: /de/node/
Disallow: /legal/sf-eula
Disallow: /legal/xpress-eula
Disallow: /analytics
Disallow: /analytics-copy
Disallow: /lp/fsa
Disallow: /fr/lp/fsa
Disallow: /de/lp/fsa
Disallow: /lp/cxo
Disallow: /ja/lp/cxo
Disallow: /fr/lp/cxo
Disallow: /de/lp/cxo
Disallow: /ko/lp/cxo
Disallow: /es-mx/lp/cxo
Disallow: /pt-br/lp/cxo
Disallow: /thank-you
Disallow: /es-mx/thank-you
Disallow: /pt-br/thank-you
Disallow: /ja/thank-you
Disallow: /de/thank-you
Disallow: /fr/thank-you
Disallow: /ko/thank-you
Disallow: /insights-test-2877
Allow: /ja/lp/fsa
Sitemap: https://www.illumio.com/sitemap.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived