NeuralCrawl

Snyk / robots.txt snapshot

← back to snyk.io · fetched 2026-06-20T14:27:25Z (7h ago) · HTTP 200 · 965 bytes · sha256 239939fe144e5a45 · raw

final URL: https://snyk.io/robots.txt

1# *
2User-agent: *
3Allow: /
4
5# *
6User-agent: *
7Disallow: /*.json$
8Disallow: /*_buildManifest.js$
9Disallow: /*_middlewareManifest.js$
10Disallow: /*_ssgManifest.js$
11Disallow: /*.js$
12
13# *
14User-agent: *
15Disallow: /advisor
16
17# Host
18Host: https://snyk.io
19
20# Sitemaps
21Sitemap: https://snyk.io/sitemap-index.xml
22Sitemap: https://snyk.io/sitemaps/sitemap-pages.xml
23Sitemap: https://snyk.io/sitemaps/sitemap-articles.xml
24Sitemap: https://snyk.io/sitemaps/sitemap-blogs.xml
25Sitemap: https://snyk.io/sitemaps/sitemap-case-studies.xml
26Sitemap: https://snyk.io/sitemaps/sitemap-contributors.xml
27Sitemap: https://snyk.io/sitemaps/sitemap-events.xml
28Sitemap: https://snyk.io/sitemaps/sitemap-podcasts.xml
29Sitemap: https://snyk.io/sitemaps/sitemap-policies.xml
30Sitemap: https://snyk.io/sitemaps/sitemap-press-releases.xml
31Sitemap: https://snyk.io/sitemaps/sitemap-videos.xml
32Sitemap: https://snyk.io/sitemaps/sitemap-misc.xml
33Sitemap: https://snyk.io/advisor/sitemap/sitemap-index.xml