NeuralCrawl

πŸ‡ΊπŸ‡Έ Qualys

qualys.com · Cybersecurity · rank #13 · Cybersecurity · live robots.txt ↗

AI crawler access (latest snapshot, 5h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 1693 bytes · sha256 23081bbac91c · raw

User-Agent: *
Allow: /
Disallow: /platform-login-ui/
Disallow: /data/
Disallow: /2020/vmdr-live/access/
Disallow: /2020/vmdr-live2/
Disallow: /2024/04/16/strategic-advisory-board/
Disallow: /2024/10/11/strategic-advisory-board/
Disallow: /community/
Disallow: /docs/2019/qsc/las-vegas/
Disallow: /docs/azure-stack-scan-report/
Disallow: /docs/qualys-2023-trurisk-threat-research-report.pdf
Disallow: /docs/version/
Disallow: /docs/whitepapers
Disallow: /lib/phpfastcache/
Disallow: /lp/2-second-visibility/
Disallow: /lp/2018/
Disallow: /lp/consolidate-your-security-stack-single-platform/
Disallow: /lp/crw/
Disallow: /lp/csam-1/
Disallow: /lp/csam-2/
Disallow: /lp/csam-3/
Disallow: /lp/dont-try-glue-siloed-security-solutions/
Disallow: /lp/end-to-end-security-less-vendors-complexity/
Disallow: /lp/free-trial/
Disallow: /lp/get-more-security/
Disallow: /lp/switch-single-vm-platform-cloud-on-prem-iot-ot/
Disallow: /partners/portal/
Disallow: /qsc/2020/virtual/agenda/
Disallow: */confirm/
Disallow: /whitepapers/2023-totalcloud-security-insights/
Disallow: /2019/analysts-investors/livestream/

Sitemap: https://www.qualys.com/sitemap.xml
Sitemap: https://www.qualys.com/sitemap-product-help.xml
Sitemap: https://www.qualys.com/sitemap-docs.xml
Sitemap: https://www.qualys.com/sitemap-release-notes.xml
Sitemap: https://www.qualys.com/sitemap-release-notes-pdf.xml
Sitemap: https://www.qualys.com/sitemap-training.xml
Sitemap: https://community.qualys.com/sitemap.xml
Sitemap: https://blog.qualys.com/sitemap.xml
Sitemap: https://notifications.qualys.com/sitemap.xml
Sitemap: https://success.qualys.com/support/s/sitemap.xml
Sitemap: https://success.qualys.com/discussions/s/sitemap.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived