NeuralCrawl

OpenAlex / robots.txt

← back to openalex.org · change detected 22d ago (2026-10-15T18:35:45Z) · modified +121 −0

What changed

Diff old (2026-07-30T16:59:37Z) → new (2026-10-15T18:35:45Z)

@@ -1,2 +1,123 @@
1+ # openalex.org robots.txt (oxjob #1689). OpenAlex data is free and CC0:
2+ # use the API or the snapshot, https://developers.openalex.org/
3+ #
4+ # A crawler obeys only the most specific group that names it (RFC 9309), so each
5+ # group repeats the rules it needs. Paths are case-sensitive. Thin records are kept
6+ # out of search indexes with a noindex meta tag, not here.
7+ # Tests: src/__tests__/robotsTxt.test.js
8+
9+ # Search engines: record pages and company pages; not faceted search or list URLs.
110  User-agent: *
11+ Disallow: /works?
12+ Disallow: /authors?
13+ Disallow: /sources?
14+ Disallow: /institutions?
15+ Disallow: /topics?
16+ Disallow: /keywords?
17+ Disallow: /publishers?
18+ Disallow: /funders?
19+ Disallow: /awards?
20+ Disallow: /concepts?
21+ Disallow: /subfields?
22+ Disallow: /fields?
23+ Disallow: /domains?
24+ Disallow: /sdgs?
25+ Disallow: /countries?
26+ Disallow: /continents?
27+ Disallow: /languages?
28+ Disallow: /types?
29+ Disallow: /source-types?
30+ Disallow: /institution-types?
31+ Disallow: /licenses?
32+ Disallow: /oa-statuses?
33+ Disallow: /indexes?
34+ Disallow: /source-lists?
35+ Disallow: /study-designs?
36+ Disallow: /locations?
37+ Disallow: /q?
38+ Disallow: /admin
39+ Disallow: /settings
40+ Disallow: /me/
41+ Disallow: /me$
42+ Disallow: /login/magic-token/
43+ Disallow: /oauth/
44+ Disallow: /verify-email
45+ Disallow: /orcid-callback
246  Allow: /
47+
48+ # AI assistants fetching a page for a person: everything a person can open.
49+ User-agent: ChatGPT-User
50+ User-agent: Claude-User
51+ User-agent: Perplexity-User
52+ User-agent: MistralAI-User
53+ Disallow: /admin
54+ Disallow: /settings
55+ Disallow: /me/
56+ Disallow: /me$
57+ Disallow: /login/magic-token/
58+ Disallow: /oauth/
59+ Disallow: /verify-email
60+ Disallow: /orcid-callback
61+ Allow: /
62+
63+ # AI training: company pages yes; record and list pages no. Short record URLs
64+ # (/W123) too; lower-case ones that share a letter with a company page (/a123)
65+ # redirect to a record path disallowed here. Google-Extended is left out on
66+ # purpose: it also controls grounding in Gemini Apps, which is assistant
67+ # traffic we want (oxjob #1689).
68+ User-agent: GPTBot
69+ User-agent: ClaudeBot
70+ User-agent: CCBot
71+ User-agent: Applebot-Extended
72+ User-agent: meta-externalagent
73+ Disallow: /works
74+ Disallow: /authors
75+ Disallow: /sources
76+ Disallow: /institutions
77+ Disallow: /topics
78+ Disallow: /keywords
79+ Disallow: /publishers
80+ Disallow: /funders
81+ Disallow: /awards
82+ Disallow: /concepts
83+ Disallow: /subfields
84+ Disallow: /fields
85+ Disallow: /domains
86+ Disallow: /sdgs
87+ Disallow: /countries
88+ Disallow: /continents
89+ Disallow: /languages
90+ Disallow: /types
91+ Disallow: /source-types
92+ Disallow: /institution-types
93+ Disallow: /licenses
94+ Disallow: /oa-statuses
95+ Disallow: /indexes
96+ Disallow: /source-lists
97+ Disallow: /study-designs
98+ Disallow: /locations
99+ Disallow: /q?
100+ Disallow: /q$
101+ Disallow: /W
102+ Disallow: /A
103+ Disallow: /S
104+ Disallow: /I
105+ Disallow: /P
106+ Disallow: /F
107+ Disallow: /C
108+ Disallow: /T
109+ Disallow: /G
110+ Disallow: /w
111+ Disallow: /f
112+ Disallow: /g
113+ Disallow: /admin
114+ Disallow: /settings
115+ Disallow: /me/
116+ Disallow: /me$
117+ Disallow: /login/magic-token/
118+ Disallow: /oauth/
119+ Disallow: /verify-email
120+ Disallow: /orcid-callback
121+ Allow: /
122+
123+ Sitemap: https://openalex.org/sitemap.xml