NeuralCrawl

Knowledge & UGC

The archive β€” every monitored site, its robots.txt history and AI-crawler stance.

All sites Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers
# Company Domain Sector robots.txt AI bots blocked Snapshots Changes Last change
1 πŸ‡ΊπŸ‡Έ Wikipedia wikipedia.org Dictionaries and Encyclopedias present 0 1 0 β€”
2 πŸ‡ΊπŸ‡Έ Reddit reddit.com Forum present 0 1 0 β€”
3 πŸ‡ΊπŸ‡Έ Stack Overflow stackoverflow.com Programming and Developer Software present 0 1 0 β€”
4 πŸ‡ΊπŸ‡Έ Stack Exchange stackexchange.com Programming and Developer Software present 0 1 0 β€”
5 πŸ‡ΊπŸ‡Έ Quora quora.com Q&A present 🤖 10 1 0 β€”
6 πŸ‡ΊπŸ‡Έ Medium medium.com Publishing present 🤖 7 1 0 β€”
7 πŸ‡ΊπŸ‡Έ Substack substack.com Publishing present 0 1 0 β€”
8 πŸ‡ΊπŸ‡Έ GitHub github.com Programming and Developer Software present 0 1 0 β€”
9 πŸ‡ΊπŸ‡Έ GitLab gitlab.com Programming and Developer Software present 0 1 0 β€”
10 πŸ‡ΊπŸ‡Έ Fandom fandom.com Dictionaries and Encyclopedias present 0 1 0 β€”
11 πŸ‡ΊπŸ‡Έ wikiHow wikihow.com Dictionaries and Encyclopedias present 🤖 24 1 0 β€”
12 πŸ‡ΊπŸ‡Έ Goodreads goodreads.com Books present 🤖 2 1 0 β€”
13 πŸ‡ΊπŸ‡Έ Genius genius.com Music present 🤖 16 1 0 β€”
14 πŸ‡ΊπŸ‡Έ Tumblr tumblr.com Social Network present 🤖 10 1 0 β€”
15 πŸ‡ΊπŸ‡Έ DEV Community dev.to Programming and Developer Software present 0 1 0 β€”
16 πŸ‡ΊπŸ‡Έ Hacker News news.ycombinator.com News and Media present 0 1 0 β€”
17 πŸ‡ΊπŸ‡Έ Wikimedia Commons commons.wikimedia.org Dictionaries and Encyclopedias present 0 1 0 β€”
18 πŸ‡ΊπŸ‡Έ Wiktionary wiktionary.org Dictionaries and Encyclopedias present 0 1 0 β€”
19 πŸ‡ΊπŸ‡Έ Internet Archive archive.org Libraries and Museums present 0 1 0 β€”
20 πŸ‡ΊπŸ‡Έ Project Gutenberg gutenberg.org Libraries and Museums present 0 1 0 β€”
21 πŸ‡ΊπŸ‡Έ Scribd scribd.com Public Records and Directories present 🤖 7 1 0 β€”
22 πŸ‡ΊπŸ‡Έ SlideShare slideshare.net Public Records and Directories present 🤖 7 1 0 β€”
23 πŸ‡ΊπŸ‡Έ Read the Docs readthedocs.org Programming and Developer Software present 0 1 0 β€”
24 πŸ‡ΊπŸ‡Έ npm npmjs.com Programming and Developer Software present 0 1 0 β€”
25 πŸ‡ΊπŸ‡Έ PyPI pypi.org Programming and Developer Software present 0 1 0 β€”
26 πŸ‡ΊπŸ‡Έ Kaggle kaggle.com Programming and Developer Software absent 0 0 0 β€”
27 πŸ‡¬πŸ‡§ Letterboxd letterboxd.com Arts and Entertainment present 🤖 13 1 0 β€”
28 πŸ‡ΊπŸ‡Έ Instructables instructables.com Arts and Entertainment present 0 1 0 β€”
29 πŸ‡ΊπŸ‡Έ Khan Academy khanacademy.org Education present 🤖 1 1 0 β€”
30 πŸ‡ΊπŸ‡Έ Coursera coursera.org Education present 🤖 1 1 0 β€”
31 πŸ‡³πŸ‡΄ W3Schools w3schools.com Programming and Developer Software present 0 1 0 β€”
32 πŸ‡ΊπŸ‡Έ MDN Web Docs developer.mozilla.org Programming and Developer Software present 0 1 0 β€”
33 πŸ‡ΊπŸ‡Έ Wikidata wikidata.org Dictionaries and Encyclopedias present 0 1 0 β€”
34 πŸ‡¨πŸ‡¦ Wattpad wattpad.com Books present 🤖 1 1 0 β€”
35 πŸ‡ΊπŸ‡Έ Archive of Our Own archiveofourown.org Books present 🤖 3 1 0 β€”
36 πŸ‡ΊπŸ‡Έ Imgur imgur.com Graphics Multimedia and Web Design present 0 1 0 β€”
37 πŸ‡ΊπŸ‡Έ Giphy giphy.com Graphics Multimedia and Web Design present 0 1 0 β€”
38 πŸ‡ΊπŸ‡Έ Notion notion.so Programming and Developer Software present 🤖 1 1 0 β€”
39 πŸ‡ΊπŸ‡Έ Figma Community figma.com Graphics Multimedia and Web Design present 🤖 11 1 0 β€”
40 πŸ‡ΊπŸ‡Έ Behance behance.net Graphics Multimedia and Web Design present 🤖 15 1 0 β€”