🇺🇸 Grok

grok.com · Top 1000 websites · rank #53 · LLM · live robots.txt ↗

AI crawler access (latest snapshot, 1h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

⛔GPTBot

⛔ChatGPT-User

✅OAI-SearchBot

⛔ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

⛔Google-Extended

⛔Applebot-Extended

⛔PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 536 bytes · sha256 6cfbb82be1e3 · raw

User-Agent: *
Allow: /
Allow: /chat$
Allow: /chat/$
Allow: /c$
Allow: /c/$
Disallow: /rest
Disallow: /share-links
Disallow: /c/
Disallow: /chat/
Disallow: /project/*
Disallow: /_worker/*

User-Agent: GPTBot
User-Agent: ChatGPT-User
User-Agent: PerplexityBot
User-Agent: ClaudeBot
User-Agent: Google-Extended
User-Agent: Applebot-Extended
Allow: /$
Allow: /chat$
Allow: /code$
Allow: /highlights
Allow: /imagine
Allow: /plans$
Allow: /project$
Allow: /tasks$
Allow: /supergrok/imagine$
Disallow: /

Sitemap: https://grok.com/sitemap.xml

Change history

initial 1h ago snapshot
- First snapshot of robots.txt archived