NeuralCrawl

Substack / robots.txt snapshot

← back to substack.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 587 bytes · sha256 bd430b80b28d5175 · raw

final URL: https://substack.com/robots.txt

1User-agent: BLEXBot
2Disallow: /
3
4User-agent: Twitterbot
5Disallow:
6
7
8
9User-agent: *
10Disallow: /action/
11Disallow: /publish
12Disallow: /sign-in
13Disallow: /channel-frame
14Disallow: /session-attribution-frame
15Disallow: /visited-surface-frame
16Disallow: /feed/private
17Disallow: /feed/podcast/*/private/*.rss
18Disallow: /subscribe
19Disallow: /lovestack/*
20Disallow: /p/*/comment/*
21Disallow: /inbox/post/*
22Disallow: /notes/post/*
23Disallow: /embed
24
25
26
27User-agent: facebookexternalhit
28Allow: /
29Allow: /subscribe
30
31
32SITEMAP: https://substack.com/sitemap.xml
33SITEMAP: https://substack.com/news_sitemap.xml
34