NeuralCrawl

🇺🇸 Snyk

snyk.io · Cybersecurity · rank #25 · Cybersecurity · live robots.txt ↗

AI crawler access (latest snapshot, 5h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot

Current robots.txt 965 bytes · sha256 239939fe144e · raw 

# *
User-agent: *
Allow: /

# *
User-agent: *
Disallow: /*.json$
Disallow: /*_buildManifest.js$
Disallow: /*_middlewareManifest.js$
Disallow: /*_ssgManifest.js$
Disallow: /*.js$

# *
User-agent: *
Disallow: /advisor

# Host
Host: https://snyk.io

# Sitemaps
Sitemap: https://snyk.io/sitemap-index.xml
Sitemap: https://snyk.io/sitemaps/sitemap-pages.xml
Sitemap: https://snyk.io/sitemaps/sitemap-articles.xml
Sitemap: https://snyk.io/sitemaps/sitemap-blogs.xml
Sitemap: https://snyk.io/sitemaps/sitemap-case-studies.xml
Sitemap: https://snyk.io/sitemaps/sitemap-contributors.xml
Sitemap: https://snyk.io/sitemaps/sitemap-events.xml
Sitemap: https://snyk.io/sitemaps/sitemap-podcasts.xml
Sitemap: https://snyk.io/sitemaps/sitemap-policies.xml
Sitemap: https://snyk.io/sitemaps/sitemap-press-releases.xml
Sitemap: https://snyk.io/sitemaps/sitemap-videos.xml
Sitemap: https://snyk.io/sitemaps/sitemap-misc.xml
Sitemap: https://snyk.io/advisor/sitemap/sitemap-index.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived