OpenAlex / robots.txt snapshot
← back to openalex.org · fetched 2026-10-16T00:44:51Z (22d ago) · HTTP 200 · 1714 bytes · sha256 e22fe131d3a3623b · raw
final URL: https://openalex.org/robots.txt
| 1 | # openalex.org robots.txt (oxjob #1689). OpenAlex data is free and CC0: |
| 2 | # use the API or the snapshot, https://developers.openalex.org/ |
| 3 | # |
| 4 | # A crawler obeys only the most specific group that names it (RFC 9309), so each |
| 5 | # group repeats the rules it needs. Paths are case-sensitive. Thin records are kept |
| 6 | # out of search indexes with a noindex meta tag, not here. |
| 7 | # Tests: src/__tests__/robotsTxt.test.js |
| 8 | |
| 9 | # Search engines and AI training crawlers (training allowed: Jason, 2026-10-15): |
| 10 | # record pages and company pages; not faceted search or list URLs. |
| 11 | User-agent: * |
| 12 | Disallow: /works? |
| 13 | Disallow: /authors? |
| 14 | Disallow: /sources? |
| 15 | Disallow: /institutions? |
| 16 | Disallow: /topics? |
| 17 | Disallow: /keywords? |
| 18 | Disallow: /publishers? |
| 19 | Disallow: /funders? |
| 20 | Disallow: /awards? |
| 21 | Disallow: /concepts? |
| 22 | Disallow: /subfields? |
| 23 | Disallow: /fields? |
| 24 | Disallow: /domains? |
| 25 | Disallow: /sdgs? |
| 26 | Disallow: /countries? |
| 27 | Disallow: /continents? |
| 28 | Disallow: /languages? |
| 29 | Disallow: /types? |
| 30 | Disallow: /source-types? |
| 31 | Disallow: /institution-types? |
| 32 | Disallow: /licenses? |
| 33 | Disallow: /oa-statuses? |
| 34 | Disallow: /indexes? |
| 35 | Disallow: /source-lists? |
| 36 | Disallow: /study-designs? |
| 37 | Disallow: /locations? |
| 38 | Disallow: /q? |
| 39 | Disallow: /admin |
| 40 | Disallow: /settings |
| 41 | Disallow: /me/ |
| 42 | Disallow: /me$ |
| 43 | Disallow: /login/magic-token/ |
| 44 | Disallow: /oauth/ |
| 45 | Disallow: /verify-email |
| 46 | Disallow: /orcid-callback |
| 47 | Allow: / |
| 48 | |
| 49 | # AI assistants fetching a page for a person: everything a person can open. |
| 50 | User-agent: ChatGPT-User |
| 51 | User-agent: Claude-User |
| 52 | User-agent: Perplexity-User |
| 53 | User-agent: MistralAI-User |
| 54 | Disallow: /admin |
| 55 | Disallow: /settings |
| 56 | Disallow: /me/ |
| 57 | Disallow: /me$ |
| 58 | Disallow: /login/magic-token/ |
| 59 | Disallow: /oauth/ |
| 60 | Disallow: /verify-email |
| 61 | Disallow: /orcid-callback |
| 62 | Allow: / |
| 63 | |
| 64 | Sitemap: https://openalex.org/sitemap.xml |