Creative rights-holders
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | Getty Images |
|
Stock media | present | 0 | 1 | 0 | — |
| 2 | 🇺🇸 | Shutterstock |
|
Stock media | present | 🤖 4 | 1 | 0 | — |
| 3 | 🇺🇸 | Adobe Stock |
|
Stock media | present | 🤖 6 | 1 | 0 | — |
| 4 | 🇬🇧 | Alamy |
|
Stock media | present | 0 | 1 | 0 | — |
| 5 | 🇺🇸 | Pond5 |
|
Stock media | blocked | 0 | 0 | 0 | — |
| 6 | 🇺🇸 | Storyblocks |
|
Stock media | present | 🤖 10 | 1 | 0 | — |
| 7 | 🇺🇸 | Universal Music Group |
|
Music | present | 🤖 1 | 1 | 0 | — |
| 8 | 🇺🇸 | Sony Music |
|
Music | present | 🤖 8 | 1 | 0 | — |
| 9 | 🇺🇸 | Warner Music Group |
|
Music | present | 0 | 1 | 0 | — |
| 10 | 🇸🇪 | Spotify |
|
Technology | present | 0 | 1 | 0 | — |
| 11 | 🇩🇪 | SoundCloud |
|
Music | present | 🤖 7 | 1 | 0 | — |
| 12 | 🇺🇸 | Tidal |
|
Music | blocked | 0 | 0 | 0 | — |
| 13 | 🇺🇸 | Bandcamp |
|
Music | present | 🤖 10 | 1 | 0 | — |
| 14 | 🇫🇷 | Believe |
|
Music | present | 0 | 1 | 0 | — |
| 15 | 🇸🇪 | Epidemic Sound |
|
Music | present | 0 | 1 | 0 | — |
| 16 | 🇺🇸 | Penguin Random House |
|
Publishing | present | 0 | 1 | 0 | — |
| 17 | 🇺🇸 | HarperCollins |
|
Publishing | present | 0 | 1 | 0 | — |
| 18 | 🇺🇸 | Hachette Book Group |
|
Publishing | present | 0 | 1 | 0 | — |
| 19 | 🇺🇸 | Macmillan |
|
Publishing | unreachable | 0 | 0 | 0 | — |
| 20 | 🇺🇸 | Simon & Schuster |
|
Publishing | present | 0 | 1 | 0 | — |
| 21 | 🇬🇧 | Pearson |
|
Publishing | present | 0 | 1 | 0 | — |
| 22 | 🇳🇱 | Elsevier |
|
Academic publishing | present | 0 | 1 | 0 | — |
| 23 | 🇩🇪 | Springer Nature |
|
Academic publishing | present | 0 | 1 | 0 | — |
| 24 | 🇺🇸 | Wiley |
|
Academic publishing | present | 0 | 1 | 0 | — |
| 25 | 🇺🇸 | Walt Disney |
|
Media | present | 0 | 1 | 0 | — |
| 26 | 🇺🇸 | Warner Bros. Discovery |
|
Media | present | 0 | 1 | 0 | — |
| 27 | 🇺🇸 | Paramount Global |
|
Media | present | 0 | 1 | 0 | — |
| 28 | 🇺🇸 | NBCUniversal |
|
Studios | present | 0 | 1 | 0 | — |
| 29 | 🇺🇸 | Sony Pictures |
|
Studios | present | 0 | 1 | 0 | — |
| 30 | 🇺🇸 | Lionsgate |
|
Studios | present | 0 | 1 | 0 | — |
| 31 | 🇺🇸 | Condé Nast |
|
Publisher | present | 0 | 1 | 0 | — |
| 32 | 🇺🇸 | Hearst |
|
Publishing | blocked | 0 | 0 | 0 | — |
| 33 | 🇩🇪 | Axel Springer |
|
Publisher | present | 0 | 1 | 0 | — |
| 35 | 🇺🇸 | ArtStation |
|
Art | present | 0 | 1 | 0 | — |
| 36 | 🇯🇵 | Pixiv |
|
Art | present | 🤖 11 | 1 | 0 | — |
| 37 | 🇨🇦 | 500px |
|
Photography | present | 0 | 1 | 0 | — |
| 38 | 🇬🇧 | Magnum Photos |
|
Photography | blocked | 0 | 0 | 0 | — |
| 39 | 🇺🇸 | IMDb |
|
Entertainment | present | 0 | 1 | 0 | — |
| 40 | 🇺🇸 | Discogs |
|
Music | blocked | 0 | 0 | 0 | — |