IMDb / robots.txt
← back to imdb.com · change detected 1mo ago (2026-07-02T00:38:22Z) · modified +0 −32
What changed
- Removed all rules for user-agent ‘360Spider’
- Removed all rules for user-agent ‘AhrefsBot’
- Removed all rules for user-agent ‘archive.org_bot’
- Removed all rules for user-agent ‘ArchiveBot’
- Removed all rules for user-agent ‘BingPreview’
- Removed all rules for user-agent ‘BrightEdge’
- Removed all rules for user-agent ‘CriteoBot’
- Removed all rules for user-agent ‘DotBot’
- Removed all rules for user-agent ‘doubleverify’
- Removed all rules for user-agent ‘DuckDuckGo-Favicons-Bot’
- Removed all rules for user-agent ‘Flipboard’
- Removed all rules for user-agent ‘GoogleBotAdWordsDesktop’
- Removed all rules for user-agent ‘GoogleBotAdWordsMobile’
- Removed all rules for user-agent ‘GoogleWebmasterTools’
- Removed all rules for user-agent ‘ias_crawler’
- Removed all rules for user-agent ‘Iframely’
- Removed all rules for user-agent ‘InternetArchiveBot’
- Removed all rules for user-agent ‘Leikibot’
- Removed all rules for user-agent ‘proximic’
- Removed all rules for user-agent ‘Qwantify’
- Removed all rules for user-agent ‘Screaming Frog SEO Spider’
- Removed all rules for user-agent ‘SemrushBot’
- Removed all rules for user-agent ‘SeznamBot’
- Removed all rules for user-agent ‘SimilarWebBot’
- Removed all rules for user-agent ‘SkypeUriPreview’
- Removed all rules for user-agent ‘Slurp’
- Removed all rules for user-agent ‘TTD-Content’
- Removed all rules for user-agent ‘Y!J-ASR’
- Removed all rules for user-agent ‘yahoo’
- Removed all rules for user-agent ‘Yahoo Ad monitoring’
- Removed all rules for user-agent ‘YandexBot’
- Removed all rules for user-agent ‘Yeti’
Diff old (2026-06-26T23:05:56Z) → new (2026-07-02T00:38:22Z)
| @@ -17,26 +17,13 @@ | ||
| 17 | 17 | User-agent: Chrome-Lighthouse |
| 18 | 18 | User-agent: Storebot-Google |
| 19 | 19 | User-agent: Mediapartners-Google |
| 20 | - User-agent: GoogleBotAdWordsMobile | |
| 21 | - User-agent: GoogleBotAdWordsDesktop | |
| 22 | - User-agent: GoogleWebmasterTools | |
| 23 | 20 | User-agent: GoogleStructuredData |
| 24 | 21 | User-agent: Google-Read-Aloud |
| 25 | 22 | User-agent: bingbot |
| 26 | - User-agent: BingPreview | |
| 27 | 23 | User-agent: Applebot |
| 28 | 24 | User-agent: DuckDuckBot |
| 29 | - User-agent: DuckDuckGo-Favicons-Bot | |
| 30 | 25 | User-agent: Baiduspider |
| 31 | - User-agent: YandexBot | |
| 32 | 26 | User-agent: Sogou |
| 33 | - User-agent: Slurp | |
| 34 | - User-agent: yahoo | |
| 35 | - User-agent: Yeti | |
| 36 | - User-agent: Y!J-ASR | |
| 37 | - User-agent: 360Spider | |
| 38 | - User-agent: Qwantify | |
| 39 | - User-agent: SeznamBot | |
| 40 | 27 | User-agent: MojeekBot |
| 41 | 28 | User-agent: facebookexternalhit |
| 42 | 29 | User-agent: Twitterbot |
| @@ -47,30 +34,11 @@ | ||
| 47 | 34 | User-agent: Discordbot |
| 48 | 35 | User-agent: Snapchat |
| 49 | 36 | User-agent: Slackbot-LinkExpanding |
| 50 | - User-agent: Flipboard | |
| 51 | - User-agent: Iframely | |
| 52 | - User-agent: SkypeUriPreview | |
| 53 | - User-agent: TTD-Content | |
| 54 | - User-agent: doubleverify | |
| 55 | - User-agent: Yahoo Ad monitoring | |
| 56 | 37 | User-agent: AmazonAdBot |
| 57 | - User-agent: ias_crawler | |
| 58 | - User-agent: Leikibot | |
| 59 | - User-agent: proximic | |
| 60 | - User-agent: CriteoBot | |
| 61 | 38 | User-agent: Amazonbot |
| 62 | 39 | User-agent: Amzn-SearchBot |
| 63 | 40 | User-agent: Amzn-User |
| 64 | - User-agent: AhrefsBot | |
| 65 | - User-agent: SemrushBot | |
| 66 | - User-agent: BrightEdge | |
| 67 | 41 | User-agent: ClarityBot |
| 68 | - User-agent: DotBot | |
| 69 | - User-agent: SimilarWebBot | |
| 70 | - User-agent: Screaming Frog SEO Spider | |
| 71 | - User-agent: archive.org_bot | |
| 72 | - User-agent: InternetArchiveBot | |
| 73 | - User-agent: ArchiveBot | |
| 74 | 42 | User-agent: Citoid |
| 75 | 43 | Allow: / |
| 76 | 44 | Disallow: /register |