NeuralCrawl

OpenAlex / robots.txt snapshot

← back to openalex.org · fetched 2026-10-15T18:35:45Z (22d ago) · HTTP 200 · 3003 bytes · sha256 e1f32f3eaa837888 · raw

final URL: https://openalex.org/robots.txt

1# openalex.org robots.txt (oxjob #1689). OpenAlex data is free and CC0:
2# use the API or the snapshot, https://developers.openalex.org/
3#
4# A crawler obeys only the most specific group that names it (RFC 9309), so each
5# group repeats the rules it needs. Paths are case-sensitive. Thin records are kept
6# out of search indexes with a noindex meta tag, not here.
7# Tests: src/__tests__/robotsTxt.test.js
8
9# Search engines: record pages and company pages; not faceted search or list URLs.
10User-agent: *
11Disallow: /works?
12Disallow: /authors?
13Disallow: /sources?
14Disallow: /institutions?
15Disallow: /topics?
16Disallow: /keywords?
17Disallow: /publishers?
18Disallow: /funders?
19Disallow: /awards?
20Disallow: /concepts?
21Disallow: /subfields?
22Disallow: /fields?
23Disallow: /domains?
24Disallow: /sdgs?
25Disallow: /countries?
26Disallow: /continents?
27Disallow: /languages?
28Disallow: /types?
29Disallow: /source-types?
30Disallow: /institution-types?
31Disallow: /licenses?
32Disallow: /oa-statuses?
33Disallow: /indexes?
34Disallow: /source-lists?
35Disallow: /study-designs?
36Disallow: /locations?
37Disallow: /q?
38Disallow: /admin
39Disallow: /settings
40Disallow: /me/
41Disallow: /me$
42Disallow: /login/magic-token/
43Disallow: /oauth/
44Disallow: /verify-email
45Disallow: /orcid-callback
46Allow: /
47
48# AI assistants fetching a page for a person: everything a person can open.
49User-agent: ChatGPT-User
50User-agent: Claude-User
51User-agent: Perplexity-User
52User-agent: MistralAI-User
53Disallow: /admin
54Disallow: /settings
55Disallow: /me/
56Disallow: /me$
57Disallow: /login/magic-token/
58Disallow: /oauth/
59Disallow: /verify-email
60Disallow: /orcid-callback
61Allow: /
62
63# AI training: company pages yes; record and list pages no. Short record URLs
64# (/W123) too; lower-case ones that share a letter with a company page (/a123)
65# redirect to a record path disallowed here. Google-Extended is left out on
66# purpose: it also controls grounding in Gemini Apps, which is assistant
67# traffic we want (oxjob #1689).
68User-agent: GPTBot
69User-agent: ClaudeBot
70User-agent: CCBot
71User-agent: Applebot-Extended
72User-agent: meta-externalagent
73Disallow: /works
74Disallow: /authors
75Disallow: /sources
76Disallow: /institutions
77Disallow: /topics
78Disallow: /keywords
79Disallow: /publishers
80Disallow: /funders
81Disallow: /awards
82Disallow: /concepts
83Disallow: /subfields
84Disallow: /fields
85Disallow: /domains
86Disallow: /sdgs
87Disallow: /countries
88Disallow: /continents
89Disallow: /languages
90Disallow: /types
91Disallow: /source-types
92Disallow: /institution-types
93Disallow: /licenses
94Disallow: /oa-statuses
95Disallow: /indexes
96Disallow: /source-lists
97Disallow: /study-designs
98Disallow: /locations
99Disallow: /q?
100Disallow: /q$
101Disallow: /W
102Disallow: /A
103Disallow: /S
104Disallow: /I
105Disallow: /P
106Disallow: /F
107Disallow: /C
108Disallow: /T
109Disallow: /G
110Disallow: /w
111Disallow: /f
112Disallow: /g
113Disallow: /admin
114Disallow: /settings
115Disallow: /me/
116Disallow: /me$
117Disallow: /login/magic-token/
118Disallow: /oauth/
119Disallow: /verify-email
120Disallow: /orcid-callback
121Allow: /
122
123Sitemap: https://openalex.org/sitemap.xml