NeuralCrawl

IMDb / robots.txt snapshot

← back to imdb.com · fetched 2026-08-25T01:27:49Z (25d ago) · HTTP 200 · 1937 bytes · sha256 26dc3d0c65a4a323 · raw

final URL: https://www.imdb.com/robots.txt

1# The content on this site is owned by IMDb.com, Inc., an Amazon.com company
2# and is made available for your personal, non-commercial use subject
3# to our Conditions of Use here: https://www.imdb.com/conditions/
4# Use of any device, tool, or process designed to data mine or scrape the content
5# using automated means is prohibited without prior written permission from IMDb.
6# For authorized data access and licensing inquiries, see: https://www.imdb.com/licensing/
7
8User-agent: Googlebot
9User-agent: Googlebot-Image
10User-agent: Googlebot-Video
11User-agent: Google-PageRenderer
12User-agent: Google-InspectionTool
13User-agent: Google-Safety
14User-agent: Google-Site-Verification
15User-agent: AdsBot-Google
16User-agent: AdsBot-Google-Mobile
17User-agent: Chrome-Lighthouse
18User-agent: Storebot-Google
19User-agent: Mediapartners-Google
20User-agent: GoogleStructuredData
21User-agent: Google-Read-Aloud
22User-agent: bingbot
23User-agent: Applebot
24User-agent: DuckDuckBot
25User-agent: DuckAssistBot
26User-agent: Baiduspider
27User-agent: Sogou
28User-agent: MojeekBot
29User-agent: facebookexternalhit
30User-agent: Twitterbot
31User-agent: Pinterestbot
32User-agent: LinkedInBot
33User-agent: WhatsApp
34User-agent: TelegramBot
35User-agent: Discordbot
36User-agent: Snapchat
37User-agent: Slackbot-LinkExpanding
38User-agent: AmazonAdBot
39User-agent: Amazonbot
40User-agent: Amzn-SearchBot
41User-agent: Amzn-User
42User-agent: ClarityBot
43User-agent: Citoid
44Allow: /
45Disallow: /register
46Disallow: /*/register
47Disallow: /registration/
48Disallow: /*/registration/
49Disallow: /search/title/*
50Disallow: /*/search/title/*
51Disallow: /search/name/*
52Disallow: /*/search/name/*
53Disallow: /find
54Disallow: /*/find
55Disallow: /find/
56Disallow: /*/find/
57Disallow: /contribute/*
58Disallow: /*/contribute/*
59Disallow: /title/*/review/*
60Disallow: /*/title/*/review/*
61Disallow: /title/*/reviews
62Disallow: /*/title/*/reviews
63Disallow: /user/ur*/reviews
64Disallow: /*/user/ur*/reviews
65
66User-agent: *
67Disallow: /