🇺🇸 University of California Berkeley

berkeley.edu · Universities · rank #6 · University · live robots.txt ↗

AI crawler access (latest snapshot, 3h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

GPTBot

ChatGPT-User

OAI-SearchBot

ClaudeBot

Claude-User

Claude-SearchBot

anthropic-ai

Claude-Web

CCBot

Google-Extended

Applebot-Extended

PerplexityBot

Perplexity-User

Bytespider

Amazonbot

FacebookBot

meta-externalagent

meta-externalfetcher

cohere-ai

AI2Bot

Diffbot

omgili

YouBot

DuckAssistBot

MistralAI-User

PanguBot

Timpibot

Current robots.txt 191 bytes · sha256 63250a737055 · raw

Crawl-delay: 10
# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

Sitemap: https://www.berkeley.edu/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

Change history

initial 3h ago snapshot
- First snapshot of robots.txt archived