Publishers
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | The New York Times |
|
News | present | 🤖 23 | 1 | 0 | — |
| 2 | 🇺🇸 | The Washington Post |
|
News | present | 🤖 13 | 1 | 0 | — |
| 3 | 🇺🇸 | The Wall Street Journal |
|
News | present | 0 | 1 | 0 | — |
| 4 | 🇺🇸 | USA Today |
|
News | present | 🤖 27 | 1 | 0 | — |
| 5 | 🇺🇸 | Los Angeles Times |
|
News | present | 🤖 5 | 1 | 0 | — |
| 6 | 🇺🇸 | Bloomberg |
|
News & Financial | present | 🤖 20 | 1 | 0 | — |
| 7 | 🇺🇸 | Reuters |
|
Newswire | present | 0 | 1 | 0 | — |
| 8 | 🇺🇸 | The Associated Press |
|
Newswire | present | 🤖 12 | 1 | 0 | — |
| 9 | 🇺🇸 | CNN |
|
Broadcast News | present | 🤖 25 | 1 | 0 | — |
| 10 | 🇺🇸 | Fox News |
|
Broadcast News | present | 0 | 1 | 0 | — |
| 11 | 🇺🇸 | NBC News |
|
Broadcast News | present | 🤖 21 | 1 | 0 | — |
| 12 | 🇺🇸 | CBS News |
|
Broadcast News | present | 🤖 1 | 1 | 0 | — |
| 13 | 🇺🇸 | ABC News |
|
Broadcast News | present | 🤖 8 | 1 | 0 | — |
| 14 | 🇺🇸 | NPR |
|
Public Radio | present | 🤖 15 | 1 | 0 | — |
| 15 | 🇺🇸 | Politico |
|
News | present | 🤖 27 | 1 | 0 | — |
| 16 | 🇺🇸 | Axios |
|
News | present | 🤖 5 | 1 | 0 | — |
| 17 | 🇺🇸 | The Atlantic |
|
Magazine | present | 🤖 19 | 1 | 0 | — |
| 18 | 🇺🇸 | The New Yorker |
|
Magazine | present | 🤖 18 | 1 | 0 | — |
| 19 | 🇺🇸 | Wired |
|
Technology | present | 🤖 18 | 1 | 0 | — |
| 20 | 🇺🇸 | The Verge |
|
Technology | present | 🤖 23 | 1 | 0 | — |
| 21 | 🇺🇸 | TechCrunch |
|
Technology | present | 🤖 13 | 1 | 0 | — |
| 22 | 🇺🇸 | Ars Technica |
|
Technology | present | 🤖 22 | 1 | 0 | — |
| 23 | 🇺🇸 | Vox |
|
News | present | 🤖 23 | 1 | 0 | — |
| 24 | 🇺🇸 | BuzzFeed |
|
Digital Media | present | 🤖 18 | 1 | 0 | — |
| 25 | 🇺🇸 | Forbes |
|
Business | present | 🤖 13 | 1 | 0 | — |
| 26 | 🇺🇸 | Business Insider |
|
Business | present | 🤖 6 | 1 | 0 | — |
| 27 | 🇺🇸 | Time |
|
Magazine | present | 0 | 1 | 0 | — |
| 28 | 🇺🇸 | Newsweek |
|
Magazine | present | 🤖 14 | 1 | 0 | — |
| 29 | 🇺🇸 | ProPublica |
|
Investigative | present | 0 | 1 | 0 | — |
| 30 | 🇺🇸 | The Hill |
|
Politics | present | 🤖 16 | 1 | 0 | — |
| 31 | 🇺🇸 | HuffPost |
|
Digital Media | present | 🤖 22 | 1 | 0 | — |
| 32 | 🇺🇸 | Slate |
|
Digital Media | present | 🤖 12 | 1 | 0 | — |
| 33 | 🇺🇸 | The Intercept |
|
Investigative | present | 🤖 7 | 1 | 0 | — |
| 34 | 🇺🇸 | Condé Nast |
|
Publisher | present | 0 | 1 | 0 | — |
| 35 | 🇺🇸 | Gannett |
|
Publisher | present | 0 | 1 | 0 | — |
| 36 | 🇬🇧 | BBC |
|
Broadcast News | present | 🤖 18 | 1 | 0 | — |
| 37 | 🇬🇧 | The Guardian |
|
News | present | 🤖 13 | 1 | 0 | — |
| 38 | 🇬🇧 | Financial Times |
|
News & Financial | present | 🤖 15 | 1 | 0 | — |
| 39 | 🇬🇧 | The Times |
|
News | present | 0 | 1 | 0 | — |
| 40 | 🇬🇧 | The Telegraph |
|
News | present | 🤖 26 | 1 | 0 | — |
| 41 | 🇬🇧 | The Independent |
|
News | present | 0 | 1 | 0 | — |
| 42 | 🇬🇧 | Daily Mail |
|
News | present | 🤖 27 | 1 | 0 | — |
| 43 | 🇬🇧 | The Economist |
|
Magazine | present | 🤖 10 | 1 | 0 | — |
| 44 | 🇬🇧 | Sky News |
|
Broadcast News | present | 🤖 16 | 1 | 0 | — |
| 45 | 🇫🇷 | Le Monde |
|
News | present | 🤖 20 | 1 | 0 | — |
| 46 | 🇫🇷 | Le Figaro |
|
News | present | 🤖 17 | 1 | 0 | — |
| 47 | 🇫🇷 | Agence France-Presse |
|
Newswire | present | 0 | 1 | 0 | — |
| 48 | 🇩🇪 | Der Spiegel |
|
News | present | 🤖 16 | 1 | 0 | — |
| 49 | 🇩🇪 | Die Zeit |
|
News | present | 🤖 14 | 1 | 0 | — |
| 50 | 🇩🇪 | Bild |
|
News | present | 🤖 14 | 1 | 0 | — |
| 51 | 🇪🇸 | El País |
|
News | present | 🤖 7 | 1 | 0 | — |
| 52 | 🇮🇹 | Corriere della Sera |
|
News | present | 🤖 8 | 1 | 0 | — |
| 53 | 🇳🇱 | De Telegraaf |
|
News | present | 🤖 22 | 1 | 0 | — |
| 54 | 🇯🇵 | Nikkei |
|
News & Financial | present | 🤖 25 | 1 | 0 | — |
| 55 | 🇯🇵 | The Asahi Shimbun |
|
News | present | 🤖 23 | 1 | 0 | — |
| 56 | 🇭🇰 | South China Morning Post |
|
News | present | 0 | 1 | 0 | — |
| 57 | 🇨🇦 | The Globe and Mail |
|
News | present | 🤖 25 | 1 | 0 | — |
| 58 | 🇨🇦 | Toronto Star |
|
News | present | 🤖 27 | 5 | 4 | 2h ago |
| 59 | 🇦🇺 | The Sydney Morning Herald |
|
News | present | 🤖 21 | 1 | 0 | — |
| 60 | 🇮🇳 | The Times of India |
|
News | present | 🤖 1 | 1 | 0 | — |
| 61 | 🇶🇦 | Al Jazeera |
|
Broadcast News | present | 🤖 8 | 1 | 0 | — |
| 62 | 🇧🇪 | Mediahuis |
|
Publisher | blocked | 0 | 0 | 0 | — |
| 63 | 🇩🇪 | Axel Springer |
|
Publisher | present | 0 | 1 | 0 | — |
| 64 | 🇺🇸 | News Corp |
|
Publisher | present | 0 | 1 | 0 | — |