NeuralCrawl

Consensus / robots.txt snapshot

← back to consensus.app · fetched 2026-06-20T14:56:27Z (4h ago) · HTTP 200 · 358 bytes · sha256 cc983dc6ecfcb67c · raw

final URL: https://consensus.app/robots.txt

1User-agent: *
2Allow: /home/wp-content/uploads/
3Disallow: /home/wp-content/plugins/
4Disallow: /home/wp-admin/
5Disallow: /home/readme.html
6Disallow: /blog/
7Crawl-delay: 1
8
9User-agent: OAI-SearchBot
10Allow: /
11
12User-agent: ChatGPT-User
13Allow: /
14
15User-agent: PerplexityBot
16Allow: /
17
18User-agent: Perplexity-User
19Allow: /
20
21Sitemap: https://consensus.app/sitemap.xml