AI Bots ▾

AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

AI Bots

All AI bots AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

The Wall Visualizer

Datasets ▾

Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Datasets

All datasets Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Feed Trends Status About

Nationwide / robots.txt snapshot

← back to nationwide.com · fetched 2026-08-27T02:04:19Z (2mo ago) · HTTP 200 · 26 bytes · sha256 331ea9090db0c9f6 · raw

final URL: https://www.nationwide.com/robots.txt

1	User-agent: *
2	Disallow: /

NeuralCrawl was created by Elie Berreby to analyze and archive the crawling directives of major companies, governments and social networks.