NeuralCrawl

IMDb / robots.txt snapshot

← back to imdb.com · fetched 2026-07-02T00:38:22Z (1mo ago) · HTTP 200 · 1911 bytes · sha256 43f808545523d2db · raw

final URL: https://www.imdb.com/robots.txt

1# The content on this site is owned by IMDb.com, Inc., an Amazon.com company
2# and is made available for your personal, non-commercial use subject
3# to our Conditions of Use here: https://www.imdb.com/conditions/
4# Use of any device, tool, or process designed to data mine or scrape the content
5# using automated means is prohibited without prior written permission from IMDb.
6# For authorized data access and licensing inquiries, see: https://www.imdb.com/licensing/
7
8User-agent: Googlebot
9User-agent: Googlebot-Image
10User-agent: Googlebot-Video
11User-agent: Google-PageRenderer
12User-agent: Google-InspectionTool
13User-agent: Google-Safety
14User-agent: Google-Site-Verification
15User-agent: AdsBot-Google
16User-agent: AdsBot-Google-Mobile
17User-agent: Chrome-Lighthouse
18User-agent: Storebot-Google
19User-agent: Mediapartners-Google
20User-agent: GoogleStructuredData
21User-agent: Google-Read-Aloud
22User-agent: bingbot
23User-agent: Applebot
24User-agent: DuckDuckBot
25User-agent: Baiduspider
26User-agent: Sogou
27User-agent: MojeekBot
28User-agent: facebookexternalhit
29User-agent: Twitterbot
30User-agent: Pinterestbot
31User-agent: LinkedInBot
32User-agent: WhatsApp
33User-agent: TelegramBot
34User-agent: Discordbot
35User-agent: Snapchat
36User-agent: Slackbot-LinkExpanding
37User-agent: AmazonAdBot
38User-agent: Amazonbot
39User-agent: Amzn-SearchBot
40User-agent: Amzn-User
41User-agent: ClarityBot
42User-agent: Citoid
43Allow: /
44Disallow: /register
45Disallow: /*/register
46Disallow: /registration/
47Disallow: /*/registration/
48Disallow: /search/title/*
49Disallow: /*/search/title/*
50Disallow: /search/name/*
51Disallow: /*/search/name/*
52Disallow: /find
53Disallow: /*/find
54Disallow: /find/
55Disallow: /*/find/
56Disallow: /contribute/*
57Disallow: /*/contribute/*
58Disallow: /title/*/review/*
59Disallow: /*/title/*/review/*
60Disallow: /title/*/reviews
61Disallow: /*/title/*/reviews
62Disallow: /user/ur*/reviews
63Disallow: /*/user/ur*/reviews
64
65User-agent: *
66Disallow: /