NeuralCrawl

Facebook / robots.txt

← back to facebook.com · change detected 1d ago (2026-11-05T19:53:47Z) · modified +3 −0

What changed

Diff old (2026-10-18T13:46:13Z) → new (2026-11-05T19:53:47Z)

@@ -5,6 +5,9 @@
55  # All authorized user-agents listed on this page must comply with Meta’s
66  # Automated Data Collection Terms available at:
77  # https://www.facebook.com/legal/automated_data_collection_terms
8+ User-agent: __usage_ai-training
9+ Disallow: /
10+
811  User-agent: Amazonbot
912  Disallow: /
1013