SEO & AI search
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | Semrush |
|
SEO software | present | 0 | 1 | 0 | — |
| 2 | 🇸🇬 | Ahrefs |
|
SEO software | present | 0 | 1 | 0 | — |
| 3 | 🇺🇸 | Similarweb |
|
Web intelligence | present | 0 | 1 | 0 | — |
| 4 | 🇺🇸 | Moz |
|
SEO software | present | 0 | 1 | 0 | — |
| 5 | 🇺🇸 | Profound |
|
AI search analytics | present | 0 | 1 | 0 | — |
| 6 | 🇺🇸 | NeuralCrawl |
|
SEO & AI Search platform | present | 0 | 1 | 0 | — |
| 7 | 🇺🇸 |
|
Search engine | present | 0 | 1 | 0 | — | |
| 8 | 🇺🇸 | OpenAI |
|
AI Chatbots and Tools | present | 0 | 1 | 0 | — |
| 9 | 🇺🇸 | Perplexity |
|
AI Chatbots and Tools | present | 0 | 1 | 0 | — |
| 10 | 🇺🇸 | Anthropic |
|
AI Chatbots and Tools | present | 0 | 1 | 0 | — |
| 11 | 🇺🇸 | Microsoft Bing |
|
Search engine | present | 0 | 1 | 0 | — |
| 12 | 🇺🇸 | BrightEdge |
|
SEO software | present | 0 | 1 | 0 | — |
| 13 | 🇺🇸 | Brave Search |
|
Search engine | present | 0 | 1 | 0 | — |
| 14 | 🇺🇸 | DuckDuckGo |
|
Search engine | present | 0 | 1 | 0 | — |
| 15 | 🇺🇸 | You.com |
|
AI answer engine | present | 0 | 1 | 0 | — |
| 16 | 🇺🇸 | Alphabet |
|
Search engine | unreachable | 0 | 0 | 0 | — |
| 17 | 🇺🇸 | seoClarity |
|
SEO software | present | 0 | 1 | 0 | — |
| 18 | 🇫🇷 | Botify |
|
SEO software | present | 0 | 1 | 0 | — |
| 19 | 🇺🇸 | Conductor |
|
SEO software | present | 0 | 1 | 0 | — |
| 20 | 🇬🇧 | Lumar |
|
SEO crawler | present | 0 | 1 | 0 | — |
| 21 | 🇵🇱 | Surfer |
|
SEO software | present | 0 | 1 | 0 | — |
| 22 | 🇺🇸 | Clearscope |
|
SEO software | present | 0 | 1 | 0 | — |
| 23 | 🇺🇸 | Exa |
|
AI search API | present | 0 | 1 | 0 | — |
| 24 | 🇺🇸 | Tavily |
|
AI search API | present | 0 | 1 | 0 | — |
| 25 | 🇺🇸 | Phind |
|
AI answer engine | blocked | 0 | 0 | 0 | — |
| 26 | 🇺🇸 | Kagi |
|
AI search engine | present | 0 | 1 | 0 | — |
| 27 | 🇺🇸 | Yahoo Search |
|
Search engine | present | 🤖 14 | 1 | 0 | — |
| 28 | 🇨🇳 | Baidu |
|
Technology | present | 0 | 1 | 0 | — |
| 29 | 🇷🇺 | Yandex |
|
Search engine | present | 0 | 1 | 0 | — |
| 30 | 🇰🇷 | Naver |
|
Search engine | present | 0 | 1 | 0 | — |
| 31 | 🇺🇸 | Andi Search |
|
AI answer engine | present | 0 | 1 | 0 | — |
| 32 | 🇯🇵 | Felo |
|
AI answer engine | present | 0 | 1 | 0 | — |
| 33 | 🇺🇸 | Komo |
|
AI answer engine | present | 0 | 1 | 0 | — |
| 34 | 🇫🇷 | Oncrawl |
|
SEO crawler | present | 0 | 1 | 0 | — |
| 35 | 🇬🇧 | Screaming Frog |
|
SEO crawler | present | 0 | 1 | 0 | — |
| 36 | 🇬🇧 | Sitebulb |
|
SEO crawler | present | 0 | 1 | 0 | — |
| 37 | 🇬🇧 | Majestic |
|
SEO intelligence | present | 0 | 1 | 0 | — |
| 38 | 🇺🇸 | SerpApi |
|
Search API | present | 0 | 1 | 0 | — |
| 39 | 🇺🇸 | Algolia |
|
Site search | present | 0 | 1 | 0 | — |
| 40 | 🇨🇦 | Coveo |
|
Enterprise search | unreachable | 0 | 0 | 0 | — |
| 41 | 🇺🇸 | Elastic |
|
Enterprise search | present | 0 | 1 | 0 | — |
| 42 | 🇺🇸 | Schema.org |
|
Structured data standard | present | 0 | 1 | 0 | — |
| 43 | 🇺🇸 | Common Crawl |
|
Web corpus | present | 0 | 1 | 0 | — |
| 44 | 🇺🇸 | Diffbot |
|
Structured web data | present | 0 | 1 | 0 | — |
| 45 | 🇮🇱 | Webz.io |
|
Web data feeds | present | 0 | 1 | 0 | — |
| 46 | 🇺🇸 | Waldo |
|
AI research search | absent | 0 | 0 | 0 | — |
| 47 | 🇺🇸 | Consensus |
|
AI research search | present | 0 | 1 | 0 | — |
| 48 | 🇺🇸 | Arc Search |
|
AI browser search | present | 0 | 1 | 0 | — |
| 49 | 🇺🇸 | Poe |
|
AI chat aggregation | present | 0 | 1 | 0 | — |
| 50 | 🇩🇪 | Ecosia |
|
Search engine | present | 0 | 1 | 0 | — |
| 51 | 🇫🇷 | Qwant |
|
Search engine | present | 0 | 1 | 0 | — |
| 52 | 🇳🇱 | Startpage |
|
Search engine | present | 0 | 1 | 0 | — |
| 53 | 🇬🇧 | Mojeek |
|
Search engine | present | 0 | 1 | 0 | — |