AI labs

The archive — every monitored site, its robots.txt history and AI-crawler stance.

#		Company	Domain	Sector	robots.txt	AI bots blocked	Snapshots	Changes	Last change
1	🇺🇸	OpenAI	openai.com	LLM	present	0	1	0	—
2	🇺🇸	Anthropic	anthropic.com	LLM	present	0	1	0	—
3	🇺🇸	Google DeepMind	deepmind.google	LLM	present	0	1	0	—
4	🇺🇸	Meta AI	ai.meta.com	LLM	present	0	1	0	—
5	🇫🇷	Mistral AI	mistral.ai	LLM	present	0	1	0	—
6	🇺🇸	xAI	x.ai	LLM	present	0	1	0	—
7	🇨🇦	Cohere	cohere.com	LLM	present	0	1	0	—
8	🇺🇸	Perplexity	perplexity.ai	LLM	present	0	1	0	—
9	🇺🇸	Hugging Face	huggingface.co	LLM	present	0	1	0	—
10	🇬🇧	Stability AI	stability.ai	LLM	present	0	1	0	—
11	🇺🇸	Midjourney	midjourney.com	LLM	present	🤖 13	1	0	—
12	🇺🇸	Runway	runwayml.com	LLM	present	0	1	0	—
14	🇺🇸	Character.AI	character.ai	LLM	present	0	1	0	—
15	🇺🇸	Inflection AI	inflection.ai	LLM	present	0	1	0	—
16	🇺🇸	Together AI	together.ai	LLM	present	🤖 1	1	0	—
17	🇩🇪	Aleph Alpha	aleph-alpha.com	LLM	present	0	1	0	—
18	🇮🇱	AI21 Labs	ai21.com	LLM	present	🤖 8	1	0	—
19	🇺🇸	Databricks	databricks.com	LLM	present	0	1	0	—
20	🇺🇸	Scale AI	scale.com	LLM	present	0	1	0	—
21	🇺🇸	Replicate	replicate.com	LLM	present	0	1	0	—
22	🇺🇸	Groq	groq.com	LLM	present	0	1	0	—
23	🇺🇸	Cerebras	cerebras.ai	LLM	present	0	1	0	—
24	🇨🇳	DeepSeek	deepseek.com	LLM	absent	0	0	0	—
25	🇨🇳	01.AI	01.ai	LLM	present	0	1	0	—
26	🇺🇸	Reka	reka.ai	LLM	present	0	1	0	—
27	🇺🇸	Contextual AI	contextual.ai	LLM	present	0	1	0	—
28	🇺🇸	Writer	writer.com	LLM	present	0	1	0	—
29	🇺🇸	Jasper	jasper.ai	LLM	present	0	1	0	—
30	🇺🇸	Suno	suno.com	LLM	present	0	1	0	—
31	🇺🇸	Udio	udio.com	LLM	present	0	1	0	—
32	🇺🇸	Luma AI	lumalabs.ai	LLM	present	0	1	0	—
33	🇺🇸	Pika	pika.art	LLM	present	0	1	0	—
34	🇺🇸	LangChain	langchain.com	LLM	present	0	1	0	—
35	🇺🇸	Pinecone	pinecone.io	LLM	present	0	1	0	—
36	🇳🇱	Weaviate	weaviate.io	LLM	present	0	1	0	—
37	🇺🇸	OpenRouter	openrouter.ai	LLM	present	0	1	0	—
38	🇺🇸	Anysphere (Cursor)	cursor.com	LLM	present	0	1	0	—
39	🇺🇸	ElevenLabs	elevenlabs.io	LLM	present	0	1	0	—