X (Twitter) / robots.txt
← back to x.com · change detected 1mo ago (2026-08-13T21:57:57Z) · modified +18 −35
What changed
- Added rules for user-agent ‘meta-externalads’
- 🤖 AIAdded rules for AI crawler meta-externalagent (Meta)
- 🤖 AIAdded rules for AI crawler meta-externalfetcher (Meta)
- 🤖 AIAdded rules for AI crawler Meta-WebIndexer (Meta)
Diff old (2026-08-06T01:10:59Z) → new (2026-08-13T21:57:57Z)
| @@ -1,6 +1,8 @@ | ||
| 1 | - # Google Search Engine Robot | |
| 2 | - # ========================== | |
| 1 | + # Google / Bing Search Engine Robots | |
| 2 | + # ================================== | |
| 3 | + # Shared group so Googlebot and Bingbot always get the same rules. | |
| 3 | 4 | User-agent: Googlebot |
| 5 | + User-agent: Bingbot | |
| 4 | 6 | |
| 5 | 7 | Allow: /*?s= |
| 6 | 8 | Allow: /*?t= |
| @@ -27,39 +29,6 @@ | ||
| 27 | 29 | |
| 28 | 30 | # Only `*` and `$` are metacharacters here (RFC 9309); regex classes match |
| 29 | 31 | # literally. /photo is anchored so /status/*/photo/1 media pages stay open. |
| 30 | - Disallow: /*/status/*/likes | |
| 31 | - Disallow: /*/status/*/retweets | |
| 32 | - Disallow: /*/likes | |
| 33 | - Disallow: /*/likes? | |
| 34 | - Disallow: /*/media | |
| 35 | - Disallow: /*/media? | |
| 36 | - Disallow: /*/photo$ | |
| 37 | - Disallow: /*/photo? | |
| 38 | - Allow: /*? | |
| 39 | - | |
| 40 | - User-agent: Bingbot | |
| 41 | - | |
| 42 | - Allow: /*?s= | |
| 43 | - Allow: /*?t= | |
| 44 | - Allow: /*?ref_src= | |
| 45 | - Allow: /hashtag/*?src= | |
| 46 | - Allow: /search?q=%23 | |
| 47 | - Allow: /i/api/ | |
| 48 | - Allow: /i/api/* | |
| 49 | - Disallow: /*?lang=en-ss | |
| 50 | - Allow: /*?lang= | |
| 51 | - Disallow: /search/realtime | |
| 52 | - Disallow: /search/users | |
| 53 | - Disallow: /search/*/grid | |
| 54 | - | |
| 55 | - Disallow: /*/analytics | |
| 56 | - Disallow: /*/followers | |
| 57 | - Disallow: /*/following | |
| 58 | - Disallow: /*/verified_followers | |
| 59 | - | |
| 60 | - Disallow: /account/deactivated | |
| 61 | - Disallow: /settings/deactivated | |
| 62 | - | |
| 63 | 32 | Disallow: /*/status/*/likes |
| 64 | 33 | Disallow: /*/status/*/retweets |
| 65 | 34 | Disallow: /*/likes |
| @@ -111,6 +80,20 @@ | ||
| 111 | 80 | User-agent: Discordbot |
| 112 | 81 | Disallow: * |
| 113 | 82 | |
| 83 | + # Meta AI / product crawlers (product tokens; versions like /1.1 are ignored | |
| 84 | + # under RFC 9309 token matching). | |
| 85 | + User-agent: meta-webindexer | |
| 86 | + Disallow: * | |
| 87 | + | |
| 88 | + User-agent: meta-externalagent | |
| 89 | + Disallow: * | |
| 90 | + | |
| 91 | + User-agent: meta-externalads | |
| 92 | + Disallow: * | |
| 93 | + | |
| 94 | + User-agent: meta-externalfetcher | |
| 95 | + Disallow: * | |
| 96 | + | |
| 114 | 97 | # Every bot that might possibly read and respect this file |
| 115 | 98 | # ======================================================== |
| 116 | 99 | User-agent: * |