AI Bots ▾

AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

AI Bots

All AI bots AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

The Wall Visualizer

Datasets ▾

Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Datasets

All datasets Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Feed Trends Status About

CSL / robots.txt snapshot

← back to csl.com · fetched 2026-08-31T21:34:42Z (2mo ago) · HTTP 200 · 55 bytes · sha256 18a04a6977101b5a · raw

final URL: https://www.csl.com/robots.txt

1	User-agent: *
2	Sitemap: https://www.csl.com/sitemap.xml

NeuralCrawl was created by Elie Berreby to analyze and archive the crawling directives of major companies, governments and social networks.