NeuralCrawl

Semantic Scholar / robots.txt snapshot

← back to semanticscholar.org · fetched 2026-06-26T14:15:22Z (4h ago) · HTTP 200 · 574 bytes · sha256 c91d2f860bd8e576 · raw

final URL: https://www.semanticscholar.org/robots.txt

1# We are a non-profit research institute. If you would like to collaborate with us,
2# please contact us at: [email protected]
3# Or check out our public API http://api.semanticscholar.org/
4User-agent: *
5Disallow: /search
6Disallow: /error
7Disallow: /me
8Disallow: /api
9Disallow: /author/*/claim
10Disallow: /author/*?
11Disallow: /paper/*?
12Disallow: /reader/
13Allow: /paper/*?p2df
14
15Sitemap: https://www.semanticscholar.org/sitemap_author_index.xml
16Sitemap: https://www.semanticscholar.org/sitemap_paper_index.xml
17Sitemap: https://www.semanticscholar.org/sitemap_topic_index.xml