NeuralCrawl

Snyk / robots.txt snapshot

← back to snyk.io · fetched 2026-09-09T00:14:16Z (1mo ago) · HTTP 200 · 968 bytes · sha256 0d7ab160cc7b88a9 · raw

final URL: https://snyk.io/robots.txt

1# *
2User-agent: *
3Allow: /
4Content-Signal: ai-train=no, search=yes, ai-input=yes
5
6# *
7User-agent: *
8Disallow: /*.json$
9Disallow: /*_buildManifest.js$
10Disallow: /*_middlewareManifest.js$
11Disallow: /*_ssgManifest.js$
12Disallow: /*.js$
13
14# *
15User-agent: *
16Disallow: /advisor
17
18# *
19User-agent: *
20Disallow: /*?topic=
21Disallow: /*&topic=
22
23# Host
24Host: https://snyk.io
25
26# Sitemaps
27Sitemap: https://snyk.io/sitemap-index.xml
28Sitemap: https://snyk.io/sitemaps/sitemap-pages.xml
29Sitemap: https://snyk.io/sitemaps/sitemap-articles.xml
30Sitemap: https://snyk.io/sitemaps/sitemap-blogs.xml
31Sitemap: https://snyk.io/sitemaps/sitemap-case-studies.xml
32Sitemap: https://snyk.io/sitemaps/sitemap-contributors.xml
33Sitemap: https://snyk.io/sitemaps/sitemap-events.xml
34Sitemap: https://snyk.io/sitemaps/sitemap-podcasts.xml
35Sitemap: https://snyk.io/sitemaps/sitemap-policies.xml
36Sitemap: https://snyk.io/sitemaps/sitemap-press-releases.xml
37Sitemap: https://snyk.io/sitemaps/sitemap-videos.xml