NeuralCrawl

Hulu / robots.txt

← back to hulu.com · change detected 27d ago (2026-07-08T02:36:30Z) · modified +132 −207

What changed

Diff old (2026-06-26T13:23:15Z) → new (2026-07-08T02:36:30Z)

@@ -1,216 +1,141 @@
11  # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
2- #
3- # To ban all spiders from the entire site uncomment the next two lines:
4- # User-Agent: *
5- # Disallow: /
2+ #robots.txt for www.hulu.com/
63 
7- User-agent: blinkx
4+ User-agent: *
85  Disallow: /
96 
10- User-agent: blinkx_ff_spider
11- Disallow: /
7+ # SEO Bots
8+ User-agent: Googlebot
9+ User-agent: Googlebot-Video
10+ User-agent: Googlebot-Image
11+ User-agent: Googlebot-News
12+ User-agent: Storebot-Google
13+ User-agent: Google-InspectionTool
14+ User-agent: Applebot
15+ User-agent: Bingbot
16+ User-agent: BingPreview
17+ User-agent: Slurp
18+ User-agent: DuckDuckBot
19+ User-agent: Baiduspider
20+ User-agent: Baiduspider-mobile
21+ User-agent: Baiduspider-video
22+ User-agent: Baiduspider-image
23+ User-agent: NaverBot
24+ User-agent: Y!J-WSC
25+ User-agent: YJ-WSC
26+ User-agent: Yeti
27+ User-agent: Yandex
28+ User-agent: YandexBot
29+ User-agent: YandexMobileBot
30+ User-agent: YandexVideo
31+ User-agent: YandexWebmaster
32+ User-agent: YandexSitelinks
33+ User-agent: SeznamBot
34+ User-agent: PetalBot
35+ User-agent: Sogou
36+ User-agent: Qwantify
37+ User-agent: Ecosia
38+ User-agent: 360Spider
39+ User-agent: Coc Coc
40+ User-agent: Qwantbot
41+ # SEO Tools
42+ User-agent: Botify
43+ User-agent: SemrushBot
44+ User-agent: Screaming Frog SEO Spider
45+ User-agent: DataForSeoBot
46+ User-agent: ia_archiver
47+ # Social
48+ User-agent: Facebot
49+ User-agent: Twitterbot
50+ User-agent: facebookexternalhit
51+ User-agent: instagram
52+ User-agent: Threads
53+ User-agent: TikTok
54+ User-agent: Pinterest
55+ User-agent: Pinterestbot
56+ User-agent: redditbot
57+ User-agent: WhatsApp
58+ User-agent: Snapchat
59+ User-agent: TelegramBot
60+ User-agent: Viber
61+ User-agent: KakaoTalk
62+ User-agent: Line
63+ # Ad Bots
64+ User-agent: AdsBot-Google
65+ User-agent: AdsBot-Google-Mobile
66+ User-agent: AdsBot-Google-Mobile-Apps
67+ User-agent: Mediapartners-Google
68+ User-agent: Google-Ads-Creatives-Assistant
69+ User-agent: FacebookAdPreview
70+ User-agent: meta-externalads
71+ User-agent: Twitterbot
72+ User-agent: Adidxbot
73+ User-agent: Bytespider
74+ User-agent: IAS-Crawler
75+ User-agent: DoubleVerifyBot
76+ User-agent: Moatbot
77+ User-agent: TheTradeDesk
78+ User-agent: TTDCrawler
79+ User-agent: CriteoBot
80+ # AI Bots
81+ User-agent: GPTBot
82+ User-agent: ChatGPT-User
83+ User-agent: OAI-SearchBot
84+ User-agent: Google-Extended
85+ User-agent: ClaudeBot
86+ User-agent: Claude-User
87+ User-agent: Claude-SearchBot
88+ User-agent: PerplexityBot
89+ User-agent: Applebot-Extended
90+ User-agent: Meta-ExternalAgent
91+ User-agent: Meta-ExternalFetcher
92+ User-agent: Amazonbot
93+ User-agent: CCBot
94+ User-agent: Google-Agent
95+ User-agent: GrokBot
96+ User-agent: YouBot
97+ User-agent: DuckAssistBot
98+ User-agent: DeepSeekBot
99+ Allow: /
12100 
13- User-agent: bingbot
101+ Disallow: /activities
102+ Disallow: /activity_feeds
103+ Disallow: /api
104+ Disallow: /browse
105+ Disallow: /channels
106+ Disallow: /discussions
107+ Disallow: /email
108+ Disallow: /exstack
109+ Disallow: /facebook_canvas
110+ Disallow: /feed
111+ Disallow: /highest_rated
112+ Disallow: /labs
14113  Disallow: /manifest.json
114+ Disallow: /masthead
115+ Disallow: /playlists
116+ Disallow: /plus
117+ Disallow: /popular
118+ Disallow: /posts
119+ Disallow: /profiles
120+ Disallow: /r
121+ Disallow: /recent
122+ Disallow: /released
123+ Disallow: /reviews
124+ Disallow: /rules
125+ Disallow: /search
126+ Disallow: /social
127+ Disallow: /start
128+ Disallow: /support
129+ Disallow: /*.swf
130+ Disallow: /tempo
131+ Disallow: /unsubscribe_email
132+ Disallow: /users
133+ Disallow: /videos
134+ Disallow: /watch
135+ Disallow: /widget
15136 
16- User-agent: Googlebot
17- Allow: /api/2.0/static/page_content
18- Disallow: /beta
19- Disallow: /_qos/
20- Disallow: /account/authenticate
21- Disallow: /activity_feeds/
22- Disallow: /activities/
23- Disallow: /api/2.0/
24- Disallow: /channels/
25- Disallow: /discussions/search
26- Disallow: /email$
27- Disallow: /email?
28- Disallow: /exstack/
29- Disallow: /facebook_canvas/
30- Disallow: /feed/activity/
31- Disallow: /feed/history/
32- Disallow: /feed/queue/
33- Disallow: /feed/search
34- Disallow: /feed/search/
35- Disallow: /feed/review/
36- Disallow: /highest_rated
37- Disallow: /highest_rated/
38- Disallow: /labs/caption-search
39- Disallow: /masthead$
40- Disallow: /masthead?
41- Disallow: /masthead/
42- Disallow: /playlists/flag/
43- Disallow: /plus/check_hold
44- Disallow: /plus/check_psn
45- Disallow: /plus/invite_request
46- Disallow: /plus/signup
47- Disallow: /plus/signup/
48- Disallow: /plus/vip
49- Disallow: /popular$
50- Disallow: /popular?
51- Disallow: /popular/
52- Disallow: /posts/flag/
53- Disallow: /posts/list/
54- Disallow: /posts/item/
55- Disallow: /profiles/
56- Disallow: /r/
57- Disallow: /recent$
58- Disallow: /recent?
59- Disallow: /recent/
60- Disallow: /released$
61- Disallow: /released?
62- Disallow: /released/
63- Disallow: /reviews/update_page
64- Disallow: /rules$
65- Disallow: /rules?
66- Disallow: /search?
67- Disallow: /search/
68- Disallow: /social/
69- Disallow: /support/geofilter
70- Disallow: /support/search
71- Disallow: /support/streaming
72- Disallow: /support/vote
73- Disallow: /users/
74- Disallow: /feed/
75- Disallow: /videos/expander
76- Disallow: /videos/search
77- Disallow: /videos/search/
78- Disallow: /videos/show_trailer
79- Disallow: /videos/slider
80- Disallow: /videos/transcripts/
81- Disallow: /widget/player/
82- Disallow: /browse/alphabetical/feature_films
83- Disallow: /*.swf$
84- Disallow: /unsubscribe_email/
85137 
86- User-agent: Google-Extended
87- Disallow: /
138+ # Announce Sitemap
139+ Sitemap: https://www.hulu.com/sitemap_index.xml
88140 
89- User-agent: GPTBot
90- Disallow: /
91-
92- User-agent: *
93- Allow: /api/2.0/static/page_content
94- Disallow: /_qos/
95- Disallow: /account/authenticate
96- Disallow: /activity_feeds/
97- Disallow: /activities/
98- Disallow: /api/2.0/
99- Disallow: /channels/
100- Disallow: /discussions/search
101- Disallow: /email$
102- Disallow: /email?
103- Disallow: /exstack/
104- Disallow: /facebook_canvas/
105- Disallow: /feed/activity/
106- Disallow: /feed/history/
107- Disallow: /feed/queue/
108- Disallow: /feed/search
109- Disallow: /feed/search/
110- Disallow: /feed/review/
111- Disallow: /highest_rated
112- Disallow: /highest_rated/
113- Disallow: /labs/caption-search
114- Disallow: /masthead$
115- Disallow: /masthead?
116- Disallow: /masthead/
117- Disallow: /mozart/
118- Disallow: /playlists/flag/
119- Disallow: /plus/check_hold
120- Disallow: /plus/check_psn
121- Disallow: /plus/invite_request
122- Disallow: /plus/signup
123- Disallow: /plus/signup/
124- Disallow: /plus/vip
125- Disallow: /popular$
126- Disallow: /popular?
127- Disallow: /popular/
128- Disallow: /posts/flag/
129- Disallow: /posts/list/
130- Disallow: /posts/item/
131- Disallow: /profiles/
132- Disallow: /r/
133- Disallow: /recent$
134- Disallow: /recent?
135- Disallow: /recent/
136- Disallow: /released$
137- Disallow: /released?
138- Disallow: /released/
139- Disallow: /reviews/update_page
140- Disallow: /rules$
141- Disallow: /rules?
142- Disallow: /search?
143- Disallow: /search/
144- Disallow: /social/
145- Disallow: /start/armstrong-1mo$
146- Disallow: /start/armstrong-1mo?
147- Disallow: /start/devices$
148- Disallow: /start/devices?
149- Disallow: /start/disney-hulu-bundle-reacq$
150- Disallow: /start/disney-hulu-bundle-reacq?
151- Disallow: /start/ellen$
152- Disallow: /start/ellen?
153- Disallow: /start/email-reacq-sp25$
154- Disallow: /start/email-reacq-sp25?
155- Disallow: /start/fyber$
156- Disallow: /start/fyber?
157- Disallow: /start/gci-1mo$
158- Disallow: /start/gci-1mo?
159- Disallow: /start/get1month$
160- Disallow: /start/get1month?
161- Disallow: /start/gifting$
162- Disallow: /start/gifting?
163- Disallow: /start/gifting_terms$
164- Disallow: /start/gifting_terms?
165- Disallow: /start/klay$
166- Disallow: /start/klay?
167- Disallow: /start/klay-livetv$
168- Disallow: /start/klay-livetv?
169- Disallow: /start/lg$
170- Disallow: /start/lg?
171- Disallow: /start/lgtv$
172- Disallow: /start/lgtv?
173- Disallow: /start/microsoft$
174- Disallow: /start/microsoft?
175- Disallow: /start/ps3$
176- Disallow: /start/ps3?
177- Disallow: /start/ps4$
178- Disallow: /start/ps4?
179- Disallow: /start/ps5$
180- Disallow: /start/ps5?
181- Disallow: /start/signup-dtp$
182- Disallow: /start/signup-dtp?
183- Disallow: /start/twdc-employee$
184- Disallow: /start/twdc-employee?
185- Disallow: /start/microsoft$
186- Disallow: /start/microsoft?
187- Disallow: /start/wow$
188- Disallow: /start/wow?
189- Disallow: /start/wow-1mo$
190- Disallow: /start/wow-1mo?
191- Disallow: /start/xbox$
192- Disallow: /start/xbox?
193- Disallow: /start/wow$
194- Disallow: /start/wow?
195- Disallow: /start/viziotvsmartcast$
196- Disallow: /start/viziotvsmartcast?
197- Disallow: /support/geofilter
198- Disallow: /support/search
199- Disallow: /support/streaming
200- Disallow: /support/vote
201- Disallow: /tempo/
202- Disallow: /tempo-fallback/
203- Disallow: /users/
204- Disallow: /feed/
205- Disallow: /videos/expander
206- Disallow: /videos/search
207- Disallow: /videos/search/
208- Disallow: /videos/show_trailer
209- Disallow: /videos/slider
210- Disallow: /videos/transcripts/
211- Disallow: /widget/player/
212- Disallow: /browse/alphabetical/feature_films
213- Disallow: /*.swf$
214- Disallow: /unsubscribe_email
215-
216- Sitemap: https://www.hulu.com/sitemap_index.xml
141+ # Source: site-static