Knowledge & UGC
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | Wikipedia |
|
Dictionaries and Encyclopedias | present | 0 | 1 | 0 | — |
| 2 | 🇺🇸 |
|
Forum | present | 0 | 1 | 0 | — | |
| 3 | 🇺🇸 | Stack Overflow |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 4 | 🇺🇸 | Stack Exchange |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 5 | 🇺🇸 | Quora |
|
Q&A | present | 🤖 10 | 1 | 0 | — |
| 6 | 🇺🇸 | Medium |
|
Publishing | present | 🤖 7 | 1 | 0 | — |
| 7 | 🇺🇸 | Substack |
|
Publishing | present | 0 | 1 | 0 | — |
| 8 | 🇺🇸 | GitHub |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 9 | 🇺🇸 | GitLab |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 10 | 🇺🇸 | Fandom |
|
Dictionaries and Encyclopedias | present | 0 | 1 | 0 | — |
| 11 | 🇺🇸 | wikiHow |
|
Dictionaries and Encyclopedias | present | 🤖 24 | 1 | 0 | — |
| 12 | 🇺🇸 | Goodreads |
|
Books | present | 🤖 2 | 1 | 0 | — |
| 13 | 🇺🇸 | Genius |
|
Music | present | 🤖 16 | 1 | 0 | — |
| 14 | 🇺🇸 | Tumblr |
|
Social Network | present | 🤖 10 | 1 | 0 | — |
| 15 | 🇺🇸 | DEV Community |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 16 | 🇺🇸 | Hacker News |
|
News and Media | present | 0 | 1 | 0 | — |
| 17 | 🇺🇸 | Wikimedia Commons |
|
Dictionaries and Encyclopedias | present | 0 | 1 | 0 | — |
| 18 | 🇺🇸 | Wiktionary |
|
Dictionaries and Encyclopedias | present | 0 | 1 | 0 | — |
| 19 | 🇺🇸 | Internet Archive |
|
Libraries and Museums | present | 0 | 1 | 0 | — |
| 20 | 🇺🇸 | Project Gutenberg |
|
Libraries and Museums | present | 0 | 1 | 0 | — |
| 21 | 🇺🇸 | Scribd |
|
Public Records and Directories | present | 🤖 7 | 1 | 0 | — |
| 22 | 🇺🇸 | SlideShare |
|
Public Records and Directories | present | 🤖 7 | 1 | 0 | — |
| 23 | 🇺🇸 | Read the Docs |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 24 | 🇺🇸 | npm |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 25 | 🇺🇸 | PyPI |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 26 | 🇺🇸 | Kaggle |
|
Programming and Developer Software | absent | 0 | 0 | 0 | — |
| 27 | 🇬🇧 | Letterboxd |
|
Arts and Entertainment | present | 🤖 13 | 1 | 0 | — |
| 28 | 🇺🇸 | Instructables |
|
Arts and Entertainment | present | 0 | 1 | 0 | — |
| 29 | 🇺🇸 | Khan Academy |
|
Education | present | 🤖 1 | 1 | 0 | — |
| 30 | 🇺🇸 | Coursera |
|
Education | present | 🤖 1 | 1 | 0 | — |
| 31 | 🇳🇴 | W3Schools |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 32 | 🇺🇸 | MDN Web Docs |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 33 | 🇺🇸 | Wikidata |
|
Dictionaries and Encyclopedias | present | 0 | 1 | 0 | — |
| 34 | 🇨🇦 | Wattpad |
|
Books | present | 🤖 1 | 1 | 0 | — |
| 35 | 🇺🇸 | Archive of Our Own |
|
Books | present | 🤖 3 | 1 | 0 | — |
| 36 | 🇺🇸 | Imgur |
imgur.com
|
Graphics Multimedia and Web Design | present | 0 | 1 | 0 | — |
| 37 | 🇺🇸 | Giphy |
giphy.com
|
Graphics Multimedia and Web Design | present | 0 | 1 | 0 | — |
| 38 | 🇺🇸 | Notion |
|
Programming and Developer Software | present | 🤖 1 | 1 | 0 | — |
| 39 | 🇺🇸 | Figma Community |
|
Graphics Multimedia and Web Design | present | 🤖 11 | 1 | 0 | — |
| 40 | 🇺🇸 | Behance |
|
Graphics Multimedia and Web Design | present | 🤖 15 | 1 | 0 | — |