NeuralCrawl

Meta AI / robots.txt snapshot

← back to ai.meta.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 860 bytes · sha256 6f3474864249dbcb · raw

final URL: https://ai.meta.com/robots.txt

1# Notice: Collection of data on Facebook through automated means is
2# prohibited unless you have express written permission from Facebook
3# and may only be conducted for the limited purpose contained in said
4# permission.
5# See: http://www.facebook.com/apps/site_scraping_tos_terms.php
6
7User-agent: facebookexternalhit
8Allow: *
9
10User-agent: meta-externalads
11Allow: *
12
13User-agent: *
14Disallow: /*cursor=
15Disallow: /*fb_comment_id=
16Disallow: /ajax/
17Disallow: /tealium/
18Disallow: /intern/
19Disallow: /internal/
20Disallow: /login/
21Disallow: /oidc/callback/
22Disallow: /*.php
23
24User-agent: PetalBot
25Disallow: /
26
27User-agent: Scrapy
28Disallow: /
29
30User-agent: uptimerobot
31Disallow: /
32
33User-agent: viberbot
34Disallow: /
35
36User-agent: YaK
37Disallow: /
38
39User-agent: Yandex
40Disallow: /
41
42User-agent: Yeti
43Disallow: /
44
45Sitemap: https://ai.meta.com/sitemap/ai_meta_com_sitemap.xml.gz