NeuralCrawl

๐Ÿ‡ฉ๐Ÿ‡ช Schwarz Group

gruppe.schwarz · European companies · rank #16 · Retail · live robots.txt ↗

AI crawler access (latest snapshot, 4mo ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅Meta-WebIndexer
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot

Current robots.txt 74 bytes · sha256 44fe9b39c27b · raw

User-agent: *
Disallow: /cps/
Sitemap: https://gruppe.schwarz/sitemap.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived