AI Bots ▾

AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

AI Bots

All AI bots AI crawlers GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended PerplexityBot Perplexity-User Bytespider Amazonbot FacebookBot meta-externalagent Meta-WebIndexer meta-externalfetcher cohere-ai AI2Bot Diffbot omgili YouBot DuckAssistBot MistralAI-User PanguBot Timpibot Search engines Googlebot Googlebot-Image Googlebot-News Bingbot Slurp DuckDuckBot Baiduspider YandexBot Applebot Sogou SeznamBot PetalBot

The Wall Visualizer

Datasets ▾

Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Datasets

All datasets Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers

Feed Trends Status About

Mediahuis / robots.txt snapshot

← back to mediahuis.com · fetched 2026-06-21T13:29:34Z (1mo ago) · HTTP 200 · 72 bytes · sha256 339f5628f03becf6 · raw

final URL: manual:file

1	User-agent: *
2	Disallow:
3
4	Sitemap: https://www.mediahuis.com/sitemap.xml

NeuralCrawl was created by Elie Berreby to analyze and archive the crawling directives of major companies, governments and social networks.