🇫🇷 Botify

botify.com · SEO & AI search · rank #18 · SEO software · live robots.txt ↗

AI crawler access (latest snapshot, 4h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot

✅ChatGPT-User

✅OAI-SearchBot

✅ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

✅Google-Extended

✅Applebot-Extended

✅PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 426 bytes · sha256 50e7ec4941b8 · raw

User-agent: * 
Disallow: /*?category=
Disallow: /customer-categories*
Disallow: /blog-categories*
Disallow: /insight-types*

#  ____        _   _  __       
# |  _ \      | | (_)/ _|      
# | |_) | ___ | |_ _| |_ _   _ 
# |  _ < / _ \| __| |  _| | | |
# | |_) | (_) | |_| | | | |_| |
# |____/ \___/ \__|_|_|  \__, |
#                         __/ |
#                        |___/ 

Sitemap: https://www.botify.com/sitemap.xml

Change history

initial 4h ago snapshot
- First snapshot of robots.txt archived