NeuralCrawl

IMDb / robots.txt

← back to imdb.com · change detected 4d ago (2026-11-02T18:55:02Z) · modified +3 −0

What changed

Diff old (2026-08-31T21:34:42Z) → new (2026-11-02T18:55:02Z)

@@ -65,5 +65,8 @@
6565  User-agent: AmazonAdBot
6666  Allow: /
6767 
68+ User-Agent: __usage_ai-training
69+ Disallow: /
70+
6871  User-agent: *
6972  Disallow: /