AI Bots ▾

AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

AI Bots

All AI bots AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

The Wall Visualizer

Datasets ▾

Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Datasets

All datasets Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Feed Trends Status About

Surfer / robots.txt snapshot

← back to surferseo.com · fetched 2026-10-22T15:00:52Z (16d ago) · HTTP 200 · 24 bytes · sha256 e1f21c9aa28d50ca · raw

final URL: https://surferseo.com/robots.txt

1	User-Agent: *
2	Disallow:

NeuralCrawl was created by Elie Berreby to analyze and archive the crawling directives of major companies, governments and social networks.