Social Networks
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 |
|
Social Network | present | 🤖 6 | 1 | 0 | — | |
| 2 | 🇺🇸 |
instagram.com
|
Social Network | present | 🤖 6 | 1 | 0 | — | |
| 3 | 🇺🇸 |
|
Messaging | present | 🤖 6 | 1 | 0 | — | |
| 4 | 🇺🇸 | Threads |
|
Social Network | present | 🤖 6 | 1 | 0 | — |
| 5 | 🇺🇸 | YouTube |
|
Video | present | 0 | 1 | 0 | — |
| 6 | 🇺🇸 | X (Twitter) |
|
Social Network | present | 0 | 1 | 0 | — |
| 7 | 🇺🇸 |
|
Professional Network | present | 🤖 19 | 1 | 0 | — | |
| 8 | 🇨🇳 | TikTok |
|
Video | present | 🤖 17 | 1 | 0 | — |
| 9 | 🇺🇸 |
|
Forum | present | 0 | 1 | 0 | — | |
| 10 | 🇺🇸 |
|
Social Network | present | 0 | 1 | 0 | — | |
| 11 | 🇺🇸 | Snapchat |
|
Messaging | present | 🤖 23 | 1 | 0 | — |
| 12 | 🇺🇸 | Discord |
|
Messaging | present | 0 | 1 | 0 | — |
| 13 | 🇺🇸 | Twitch |
|
Video | present | 0 | 1 | 0 | — |
| 14 | 🇦🇪 | Telegram |
|
Messaging | absent | 0 | 0 | 0 | — |
| 15 | 🇺🇸 | Tumblr |
|
Social Network | present | 🤖 10 | 1 | 0 | — |
| 16 | 🇺🇸 | Medium |
|
Publishing | present | 🤖 7 | 1 | 0 | — |
| 17 | 🇺🇸 | Quora |
|
Q&A | present | 🤖 10 | 1 | 0 | — |
| 18 | 🇩🇪 | Mastodon |
|
Social Network (Fediverse) | present | 🤖 1 | 1 | 0 | — |
| 19 | 🇺🇸 | Bluesky |
|
Social Network | present | 0 | 1 | 0 | — |
| 20 | 🇺🇸 | Nextdoor |
|
Social Network | present | 🤖 1 | 1 | 0 | — |
| 21 | 🇷🇺 | VK |
|
Social Network | present | 0 | 1 | 0 | — |
| 22 | 🇨🇳 |
|
Social Network | present | 0 | 1 | 0 | — | |
| 23 | 🇨🇳 |
|
Messaging | absent | 0 | 0 | 0 | — | |
| 24 | 🇺🇸 | Signal |
|
Messaging | absent | 0 | 0 | 0 | — |
| 25 | 🇺🇸 | Flickr |
|
Photo Sharing | present | 0 | 1 | 0 | — |
| 26 | 🇺🇸 | Vimeo |
|
Video | present | 🤖 10 | 1 | 0 | — |
| 27 | 🇺🇸 | Substack |
|
Publishing | present | 0 | 1 | 0 | — |
| 28 | 🇺🇸 | Clubhouse |
|
Audio | present | 0 | 1 | 0 | — |