NeuralCrawl

IMDb / robots.txt snapshot

← back to imdb.com · fetched 2026-06-20T01:10:31Z (18h ago) · HTTP 200 · 2686 bytes · sha256 36dafb4bc0cb0c4d · raw

final URL: https://www.imdb.com/robots.txt

1# The content on this site is owned by IMDb.com, Inc., an Amazon.com company
2# and is made available for your personal, non-commercial use subject
3# to our Conditions of Use here: https://www.imdb.com/conditions/
4# Use of any device, tool, or process designed to data mine or scrape the content
5# using automated means is prohibited without prior written permission from IMDb.
6# For authorized data access and licensing inquiries, see: https://www.imdb.com/licensing/
7
8User-agent: Googlebot
9User-agent: Googlebot-Image
10User-agent: Googlebot-Video
11User-agent: Google-PageRenderer
12User-agent: Google-InspectionTool
13User-agent: Google-Safety
14User-agent: Google-Site-Verification
15User-agent: AdsBot-Google
16User-agent: AdsBot-Google-Mobile
17User-agent: Chrome-Lighthouse
18User-agent: Storebot-Google
19User-agent: Mediapartners-Google
20User-agent: GoogleBotAdWordsMobile
21User-agent: GoogleBotAdWordsDesktop
22User-agent: GoogleWebmasterTools
23User-agent: GoogleStructuredData
24User-agent: Google-Read-Aloud
25User-agent: bingbot
26User-agent: BingPreview
27User-agent: Applebot
28User-agent: DuckDuckBot
29User-agent: DuckDuckGo-Favicons-Bot
30User-agent: Baiduspider
31User-agent: YandexBot
32User-agent: Sogou
33User-agent: Slurp
34User-agent: yahoo
35User-agent: Yeti
36User-agent: Y!J-ASR
37User-agent: 360Spider
38User-agent: Qwantify
39User-agent: SeznamBot
40User-agent: MojeekBot
41User-agent: facebookexternalhit
42User-agent: Twitterbot
43User-agent: Pinterestbot
44User-agent: LinkedInBot
45User-agent: WhatsApp
46User-agent: TelegramBot
47User-agent: Discordbot
48User-agent: Snapchat
49User-agent: Slackbot-LinkExpanding
50User-agent: Flipboard
51User-agent: Iframely
52User-agent: SkypeUriPreview
53User-agent: TTD-Content
54User-agent: doubleverify
55User-agent: Yahoo Ad monitoring
56User-agent: AmazonAdBot
57User-agent: ias_crawler
58User-agent: Leikibot
59User-agent: proximic
60User-agent: CriteoBot
61User-agent: Amazonbot
62User-agent: Amzn-SearchBot
63User-agent: AhrefsBot
64User-agent: SemrushBot
65User-agent: BrightEdge
66User-agent: ClarityBot
67User-agent: DotBot
68User-agent: SimilarWebBot
69User-agent: Screaming Frog SEO Spider
70User-agent: archive.org_bot
71User-agent: InternetArchiveBot
72User-agent: ArchiveBot
73User-agent: Citoid
74Allow: /
75Disallow: /register
76Disallow: /*/register
77Disallow: /registration/
78Disallow: /*/registration/
79Disallow: /search/title/*
80Disallow: /*/search/title/*
81Disallow: /search/name/*
82Disallow: /*/search/name/*
83Disallow: /find
84Disallow: /*/find
85Disallow: /find/
86Disallow: /*/find/
87Disallow: /contribute/*
88Disallow: /*/contribute/*
89Disallow: /title/*/review/*
90Disallow: /*/title/*/review/*
91Disallow: /title/*/reviews
92Disallow: /*/title/*/reviews
93Disallow: /user/ur*/reviews
94Disallow: /*/user/ur*/reviews
95
96User-agent: *
97Disallow: /