Academic & open research
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI models
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
Academic & open research
Universities
National indices
Banks & fintech
E-commerce
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | arXiv |
|
Research repository | present | 0 | 1 | 0 | — |
| 2 | 🇺🇸 | Google Scholar |
|
Academic search | present | 0 | 1 | 0 | — |
| 3 | 🇺🇸 | PubMed / NCBI |
|
Research repository | present | 0 | 1 | 0 | — |
| 4 | 🇩🇪 | ResearchGate |
|
Research network | present | 0 | 1 | 0 | — |
| 5 | 🇺🇸 | Academia.edu |
|
Research network | present | 🤖 3 | 1 | 0 | — |
| 6 | 🇺🇸 | Semantic Scholar |
|
Academic search | present | 0 | 1 | 0 | — |
| 7 | 🇺🇸 | SSRN |
|
Preprint server | present | 🤖 3 | 1 | 0 | — |
| 8 | 🇺🇸 | bioRxiv |
|
Preprint server | present | 🤖 7 | 1 | 0 | — |
| 9 | 🇺🇸 | medRxiv |
|
Preprint server | present | 0 | 1 | 0 | — |
| 10 | 🇺🇸 | OpenAlex |
|
Open catalog | blocked | 0 | 0 | 0 | — |
| 11 | 🇨🇭 | Zenodo |
|
Research repository | present | 0 | 1 | 0 | — |
| 12 | 🇺🇸 | Open Science Framework |
|
Research repository | present | 0 | 1 | 0 | — |
| 13 | 🇬🇧 | CORE |
|
Research aggregator | present | 0 | 1 | 0 | — |
| 14 | 🇫🇷 | HAL Open Archive |
|
Research repository | redirect | 0 | 0 | 0 | — |
| 15 | 🇬🇧 | Europe PMC |
|
Research repository | present | 0 | 1 | 0 | — |
| 16 | 🇺🇸 | Crossref |
|
Scholarly metadata | present | 0 | 1 | 0 | — |
| 17 | 🇬🇧 | DOAJ |
|
Open access directory | present | 🤖 8 | 1 | 0 | — |
| 18 | 🇺🇸 | PLOS |
|
Open access journals | present | 0 | 1 | 0 | — |
| 19 | 🇺🇸 | ORCID |
|
Researcher identifiers | present | 0 | 1 | 0 | — |
| 20 | 🇩🇪 | DBLP |
|
CS bibliography | present | 0 | 1 | 0 | — |