AI models
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI models
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
Academic & open research
Universities
National indices
Banks & fintech
E-commerce
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | ChatGPT |
|
LLM | present | 🤖 8 | 7 | 6 | 11h ago |
| 2 | 🇺🇸 | Gemini |
|
LLM | present | 0 | 1 | 0 | — |
| 3 | 🇺🇸 | Claude |
|
LLM | present | 🤖 4 | 2 | 1 | 5d ago |
| 4 | 🇺🇸 | Grok |
|
LLM | present | 🤖 6 | 1 | 0 | — |
| 5 | 🇺🇸 | Perplexity |
|
AI answer engine | present | 0 | 4 | 3 | 1mo ago |
| 6 | 🇺🇸 | Character.AI |
|
LLM | present | 0 | 1 | 0 | — |
| 7 | 🇺🇸 | OpenAI |
|
Corporate site | present | 0 | 1 | 0 | — |
| 8 | 🇺🇸 | Anthropic |
|
Corporate site | present | 0 | 1 | 0 | — |
| 9 | 🇺🇸 | Google DeepMind |
|
LLM | present | 0 | 1 | 0 | — |
| 10 | 🇺🇸 | Meta AI |
|
LLM | present | 0 | 1 | 0 | — |
| 11 | 🇺🇸 | xAI |
|
LLM | present | 0 | 1 | 0 | — |
| 12 | 🇨🇳 | DeepSeek |
|
LLM | absent | 0 | 0 | 0 | — |
| 13 | 🇫🇷 | Mistral AI |
|
LLM | present | 0 | 1 | 0 | — |
| 14 | 🇺🇸 | Midjourney |
|
LLM | present | 🤖 13 | 1 | 0 | — |
| 15 | 🇺🇸 | Hugging Face |
|
LLM | present | 0 | 1 | 0 | — |
| 16 | 🇺🇸 | Anysphere (Cursor) |
|
LLM | present | 0 | 3 | 2 | 5h ago |
| 17 | 🇺🇸 | Databricks |
|
LLM | present | 0 | 1 | 0 | — |
| 18 | 🇺🇸 | ElevenLabs |
|
LLM | present | 0 | 1 | 0 | — |
| 19 | 🇺🇸 | Runway |
|
LLM | present | 0 | 2 | 1 | 12d ago |
| 20 | 🇺🇸 | Suno |
|
LLM | present | 0 | 3 | 2 | 23h ago |
| 21 | 🇬🇧 | Stability AI |
|
LLM | present | 0 | 1 | 0 | — |
| 22 | 🇨🇦 | Cohere |
|
LLM | present | 0 | 3 | 2 | 5d ago |
| 23 | 🇺🇸 | Scale AI |
|
LLM | present | 0 | 3 | 2 | 4d ago |
| 24 | 🇺🇸 | Together AI |
|
LLM | present | 🤖 1 | 1 | 0 | — |
| 25 | 🇺🇸 | OpenRouter |
|
LLM | present | 0 | 2 | 1 | 24d ago |
| 26 | 🇺🇸 | Groq |
|
LLM | present | 0 | 2 | 1 | 5h ago |
| 27 | 🇺🇸 | Inflection AI |
|
LLM | present | 0 | 1 | 0 | — |
| 28 | 🇩🇪 | Aleph Alpha |
|
LLM | present | 0 | 1 | 0 | — |
| 29 | 🇮🇱 | AI21 Labs |
|
LLM | present | 🤖 8 | 2 | 1 | 1mo ago |
| 30 | 🇺🇸 | Replicate |
|
LLM | present | 0 | 1 | 0 | — |
| 31 | 🇺🇸 | Jasper |
|
LLM | present | 0 | 1 | 0 | — |
| 32 | 🇺🇸 | Writer |
|
LLM | present | 0 | 1 | 0 | — |
| 33 | 🇺🇸 | LangChain |
|
LLM | present | 0 | 1 | 0 | — |
| 34 | 🇺🇸 | Pinecone |
|
LLM | present | 0 | 2 | 1 | 1mo ago |
| 35 | 🇳🇱 | Weaviate |
|
LLM | present | 0 | 1 | 0 | — |
| 36 | 🇺🇸 | Cerebras |
|
LLM | present | 0 | 1 | 0 | — |
| 37 | 🇺🇸 | Luma AI |
|
LLM | present | 0 | 2 | 1 | 1mo ago |
| 38 | 🇺🇸 | Pika |
|
LLM | present | 0 | 1 | 0 | — |
| 39 | 🇺🇸 | Udio |
|
LLM | present | 0 | 1 | 0 | — |
| 40 | 🇺🇸 | Reka |
|
LLM | present | 0 | 1 | 0 | — |
| 41 | 🇺🇸 | Contextual AI |
|
LLM | present | 0 | 9 | 8 | 15d ago |
| 42 | 🇨🇳 | 01.AI |
|
LLM | present | 0 | 2 | 1 | 29d ago |