NeuralCrawl

🇺🇸 Hulu

hulu.com · Top 1000 websites · rank #98 · Web · live robots.txt ↗

AI crawler access (latest snapshot, 1h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

⛔GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
⛔Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot

Current robots.txt 5157 bytes · sha256 3e943561df9a · raw 

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /

User-agent: blinkx
Disallow: /

User-agent: blinkx_ff_spider
Disallow: /

User-agent: bingbot
Disallow: /manifest.json

User-agent: Googlebot
Allow: /api/2.0/static/page_content
Disallow: /beta
Disallow: /_qos/
Disallow: /account/authenticate
Disallow: /activity_feeds/
Disallow: /activities/
Disallow: /api/2.0/
Disallow: /channels/
Disallow: /discussions/search
Disallow: /email$
Disallow: /email?
Disallow: /exstack/
Disallow: /facebook_canvas/
Disallow: /feed/activity/
Disallow: /feed/history/
Disallow: /feed/queue/
Disallow: /feed/search
Disallow: /feed/search/
Disallow: /feed/review/
Disallow: /highest_rated
Disallow: /highest_rated/
Disallow: /labs/caption-search
Disallow: /masthead$
Disallow: /masthead?
Disallow: /masthead/
Disallow: /playlists/flag/
Disallow: /plus/check_hold
Disallow: /plus/check_psn
Disallow: /plus/invite_request
Disallow: /plus/signup
Disallow: /plus/signup/
Disallow: /plus/vip
Disallow: /popular$
Disallow: /popular?
Disallow: /popular/
Disallow: /posts/flag/
Disallow: /posts/list/
Disallow: /posts/item/
Disallow: /profiles/
Disallow: /r/
Disallow: /recent$
Disallow: /recent?
Disallow: /recent/
Disallow: /released$
Disallow: /released?
Disallow: /released/
Disallow: /reviews/update_page
Disallow: /rules$
Disallow: /rules?
Disallow: /search?
Disallow: /search/
Disallow: /social/
Disallow: /support/geofilter
Disallow: /support/search
Disallow: /support/streaming
Disallow: /support/vote
Disallow: /users/
Disallow: /feed/
Disallow: /videos/expander
Disallow: /videos/search
Disallow: /videos/search/
Disallow: /videos/show_trailer
Disallow: /videos/slider
Disallow: /videos/transcripts/
Disallow: /widget/player/
Disallow: /browse/alphabetical/feature_films
Disallow: /*.swf$
Disallow: /unsubscribe_email/

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /api/2.0/static/page_content
Disallow: /_qos/
Disallow: /account/authenticate
Disallow: /activity_feeds/
Disallow: /activities/
Disallow: /api/2.0/
Disallow: /channels/
Disallow: /discussions/search
Disallow: /email$
Disallow: /email?
Disallow: /exstack/
Disallow: /facebook_canvas/
Disallow: /feed/activity/
Disallow: /feed/history/
Disallow: /feed/queue/
Disallow: /feed/search
Disallow: /feed/search/
Disallow: /feed/review/
Disallow: /highest_rated
Disallow: /highest_rated/
Disallow: /labs/caption-search
Disallow: /masthead$
Disallow: /masthead?
Disallow: /masthead/
Disallow: /mozart/
Disallow: /playlists/flag/
Disallow: /plus/check_hold
Disallow: /plus/check_psn
Disallow: /plus/invite_request
Disallow: /plus/signup
Disallow: /plus/signup/
Disallow: /plus/vip
Disallow: /popular$
Disallow: /popular?
Disallow: /popular/
Disallow: /posts/flag/
Disallow: /posts/list/
Disallow: /posts/item/
Disallow: /profiles/
Disallow: /r/
Disallow: /recent$
Disallow: /recent?
Disallow: /recent/
Disallow: /released$
Disallow: /released?
Disallow: /released/
Disallow: /reviews/update_page
Disallow: /rules$
Disallow: /rules?
Disallow: /search?
Disallow: /search/
Disallow: /social/
Disallow: /start/armstrong-1mo$
Disallow: /start/armstrong-1mo?
Disallow: /start/devices$
Disallow: /start/devices?
Disallow: /start/disney-hulu-bundle-reacq$
Disallow: /start/disney-hulu-bundle-reacq?
Disallow: /start/ellen$
Disallow: /start/ellen?
Disallow: /start/email-reacq-sp25$
Disallow: /start/email-reacq-sp25?
Disallow: /start/fyber$
Disallow: /start/fyber?
Disallow: /start/gci-1mo$
Disallow: /start/gci-1mo?
Disallow: /start/get1month$
Disallow: /start/get1month?
Disallow: /start/gifting$
Disallow: /start/gifting?
Disallow: /start/gifting_terms$
Disallow: /start/gifting_terms?
Disallow: /start/klay$
Disallow: /start/klay?
Disallow: /start/klay-livetv$
Disallow: /start/klay-livetv?
Disallow: /start/lg$
Disallow: /start/lg?
Disallow: /start/lgtv$
Disallow: /start/lgtv?
Disallow: /start/microsoft$
Disallow: /start/microsoft?
Disallow: /start/ps3$
Disallow: /start/ps3?
Disallow: /start/ps4$
Disallow: /start/ps4?
Disallow: /start/ps5$
Disallow: /start/ps5?
Disallow: /start/signup-dtp$
Disallow: /start/signup-dtp?
Disallow: /start/twdc-employee$
Disallow: /start/twdc-employee?
Disallow: /start/microsoft$
Disallow: /start/microsoft?
Disallow: /start/wow$
Disallow: /start/wow?
Disallow: /start/wow-1mo$
Disallow: /start/wow-1mo?
Disallow: /start/xbox$
Disallow: /start/xbox?
Disallow: /start/wow$
Disallow: /start/wow?
Disallow: /start/viziotvsmartcast$
Disallow: /start/viziotvsmartcast?
Disallow: /support/geofilter
Disallow: /support/search
Disallow: /support/streaming
Disallow: /support/vote
Disallow: /tempo/
Disallow: /tempo-fallback/
Disallow: /users/
Disallow: /feed/
Disallow: /videos/expander
Disallow: /videos/search
Disallow: /videos/search/
Disallow: /videos/show_trailer
Disallow: /videos/slider
Disallow: /videos/transcripts/
Disallow: /widget/player/
Disallow: /browse/alphabetical/feature_films
Disallow: /*.swf$
Disallow: /unsubscribe_email

Sitemap: https://www.hulu.com/sitemap_index.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived