NeuralCrawl

πŸ‡ΊπŸ‡Έ Disneyplus

disneyplus.com · Top 1000 websites · rank #97 · Web · live robots.txt ↗

AI crawler access (latest snapshot, 1h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 2794 bytes · sha256 ea1fedfeaf8c · raw

#robots.txt for www.disneyplus.com/

User-agent: *
Disallow: /

# SEO Bots
User-agent: Googlebot
User-agent: Googlebot-Video
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Storebot-Google
User-agent: Google-InspectionTool
User-agent: Applebot
User-agent: Bingbot
User-agent: BingPreview
User-agent: Slurp
User-agent: DuckDuckBot
User-agent: Baiduspider
User-agent: Baiduspider-mobile
User-agent: Baiduspider-video
User-agent: Baiduspider-image
User-agent: NaverBot
User-agent: Y!J-WSC
User-agent: YJ-WSC
User-agent: Yeti
User-agent: Yandex
User-agent: YandexBot
User-agent: YandexMobileBot
User-agent: YandexVideo
User-agent: YandexWebmaster
User-agent: YandexSitelinks
User-agent: SeznamBot
User-agent: PetalBot
User-agent: Sogou
User-agent: Qwantify
User-agent: Ecosia
User-agent: 360Spider
User-agent: Coc Coc
User-agent: Qwantbot
# SEO Tools
User-agent: Botify
User-agent: SemrushBot
User-agent: Screaming Frog SEO Spider
User-agent: DataForSeoBot
User-agent: ia_archiver
# Social
User-agent: Facebot
User-agent: Twitterbot
User-agent: facebookexternalhit
User-agent: instagram
User-agent: Threads
User-agent: TikTok
User-agent: Pinterest
User-agent: Pinterestbot
User-agent: redditbot
User-agent: WhatsApp
User-agent: Snapchat
User-agent: TelegramBot
User-agent: Viber
User-agent: KakaoTalk
User-agent: Line
# Ad Bots
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: Mediapartners-Google
User-agent: Google-Ads-Creatives-Assistant
User-agent: FacebookAdPreview
User-agent: meta-externalads
User-agent: Twitterbot
User-agent: Adidxbot
User-agent: Bytespider
User-agent: IAS-Crawler
User-agent: DoubleVerifyBot
User-agent: Moatbot
User-agent: TheTradeDesk
User-agent: TTDCrawler
User-agent: CriteoBot
# AI Bots
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: CCBot
User-agent: Google-Agent
User-agent: GrokBot
User-agent: YouBot
User-agent: DuckAssistBot
User-agent: DeepSeekBot
Allow: /

# Auth Account Related Pages
Disallow: /home
Disallow: /*/home
Disallow: /episode
Disallow: /*/episode
Disallow: /watch
Disallow: /*/watch
Disallow: /billing
Disallow: /*/billing
Disallow: /search
Disallow: /*/search
Disallow: /account
Disallow: /*/account
Disallow: /app-settings
Disallow: /*/app-settings
Disallow: /enter-password
Disallow: /*/enter-password
Disallow: /create-password
Disallow: /*/create-password
Disallow: /welcome/responder

# Announce Sitemap
Sitemap: https://www.disneyplus.com/sitemap.xml

# Source: static-root-proxy

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived