🇺🇸 Diffbot

diffbot.com · SEO & AI search · rank #44 · Structured web data · live robots.txt ↗

AI crawler access (latest snapshot, 4h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot

✅ChatGPT-User

✅OAI-SearchBot

✅ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

✅Google-Extended

✅Applebot-Extended

✅PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 757 bytes · sha256 6dfb08e98993 · raw

#     ____          _                        _           _        _                                         
#    ||o o|     ___| |_ ___  _ __  _ __ ___ | |__   ___ | |_ __ _| |__  _   _ ___  ___   ___ ___  _ __ ___  
#    ||===|    / __| __/ _ \| '_ \| '__/ _ \| '_ \ / _ \| __/ _` | '_ \| | | / __|/ _ \ / __/ _ \| '_ ` _ \ 
#  .-.`---'-.  \__ \ || (_) | |_) | | | (_) | |_) | (_) | || (_| | |_) | |_| \__ \  __/| (_| (_) | | | | | |
#  | | o .o |  |___/\__\___/| .__/|_|  \___/|_.__/ \___/ \__\__,_|_.__/ \__,_|___/\___(_)___\___/|_| |_| |_|
#  | | o:.o |               |_|                                                                             
#  | |      |
#  `-".-.-.-'    This website is officially robot abuse free.
#  _| | : |_
# (oOoOo)_)_)

Change history

initial 4h ago snapshot
- First snapshot of robots.txt archived