SoundCloud / robots.txt
← back to soundcloud.com · change detected 1mo ago (2026-06-26T04:48:20Z) · modified +29 −60
What changed
- Blocked user-agent ‘Brightbot’ entirely
- 🤖 AIBlocked omgili (Webz.io) entirely
- Blocked user-agent ‘omgilibot’ entirely
- 🤖 AIUnblocked Amazonbot (Amazon) - rules removed
- 🤖 AIUnblocked Applebot-Extended (Apple) - rules removed
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- 🤖 AIUnblocked ChatGPT-User (OpenAI) - rules removed
- 🤖 AIUnblocked Claude-Web (Anthropic) - rules removed
- 🤖 AIUnblocked cohere-ai (Cohere) - rules removed
- 🤖 AIUnblocked OAI-SearchBot (OpenAI) - rules removed
- 🤖 AIUnblocked PerplexityBot (Perplexity) - rules removed
- Added Disallow: (empty) for *
- Removed Disallow: /*? for *
- Removed Disallow: /messages for *
- Removed Disallow: /search for *
- Removed Disallow: /settings for *
- Removed Disallow: /stream for *
- Removed Disallow: /upload for *
- Removed Disallow: /you/ for *
- 🤖 AIRemoved Allow: /$ for anthropic-ai
- 🤖 AIRemoved Allow: /accessibility-statement for anthropic-ai
- 🤖 AIRemoved Allow: /community-guidelines for anthropic-ai
- 🤖 AIRemoved Allow: /company for anthropic-ai
- 🤖 AIRemoved Allow: /discover for anthropic-ai
- 🤖 AIRemoved Allow: /download for anthropic-ai
- 🤖 AIRemoved Allow: /getstarted for anthropic-ai
- 🤖 AIRemoved Allow: /go for anthropic-ai
- 🤖 AIRemoved Allow: /imprint for anthropic-ai
- 🤖 AIRemoved Allow: /jobs for anthropic-ai
- 🤖 AIRemoved Allow: /pro for anthropic-ai
- 🤖 AIRemoved Allow: /sitemap for anthropic-ai
- 🤖 AIRemoved Allow: /sitemapIndex for anthropic-ai
- 🤖 AIRemoved Allow: /stories for anthropic-ai
- 🤖 AIRemoved Allow: /terms-of-use for anthropic-ai
- 🤖 AIRemoved Allow: /topic for anthropic-ai
- 🤖 AIRemoved Allow: /transparency-reports for anthropic-ai
- 🤖 AIRemoved Allow: /$ for CCBot
- 🤖 AIRemoved Allow: /accessibility-statement for CCBot
- 🤖 AIRemoved Allow: /community-guidelines for CCBot
- 🤖 AIRemoved Allow: /company for CCBot
- 🤖 AIRemoved Allow: /discover for CCBot
- 🤖 AIRemoved Allow: /download for CCBot
- 🤖 AIRemoved Allow: /getstarted for CCBot
- 🤖 AIRemoved Allow: /go for CCBot
- 🤖 AIRemoved Allow: /imprint for CCBot
- 🤖 AIRemoved Allow: /jobs for CCBot
- 🤖 AIRemoved Allow: /pro for CCBot
- 🤖 AIRemoved Allow: /sitemap for CCBot
- 🤖 AIRemoved Allow: /sitemapIndex for CCBot
- 🤖 AIRemoved Allow: /stories for CCBot
- ... and 71 more change(s)
Diff old (2026-06-25T22:38:12Z) → new (2026-06-26T04:48:20Z)
| @@ -1,65 +1,34 @@ | ||
| 1 | - # ============================================================================= | |
| 2 | - # robots.txt for soundcloud.com | |
| 3 | - # Updated: 2026-05-05 | |
| 4 | - # ============================================================================= | |
| 1 | + # This is for the m.soundcloud.com hostname | |
| 5 | 2 | |
| 6 | - # AI Crawlers: editorial only, no UGC training | |
| 3 | + User-Agent: * | |
| 4 | + Disallow: | |
| 5 | + | |
| 7 | 6 | User-Agent: anthropic-ai |
| 8 | - User-Agent: ClaudeBot | |
| 9 | - User-Agent: Claude-Web | |
| 10 | - User-Agent: GPTBot | |
| 11 | - User-Agent: ChatGPT-User | |
| 12 | - User-Agent: OAI-SearchBot | |
| 13 | - User-Agent: CCBot | |
| 14 | - User-Agent: PerplexityBot | |
| 15 | - User-Agent: Google-Extended | |
| 16 | - User-Agent: Applebot-Extended | |
| 17 | - User-Agent: Bytespider | |
| 18 | - User-Agent: Amazonbot | |
| 19 | - User-Agent: Meta-ExternalAgent | |
| 20 | - User-Agent: cohere-ai | |
| 21 | - | |
| 22 | - # Homepage | |
| 23 | - Allow: /$ | |
| 24 | - | |
| 25 | - # Platform: Legal & Policy | |
| 26 | - Allow: /terms-of-use | |
| 27 | - Allow: /community-guidelines | |
| 28 | - Allow: /transparency-reports | |
| 29 | - Allow: /accessibility-statement | |
| 30 | - Allow: /imprint | |
| 31 | - | |
| 32 | - # Platform: Discovery & Editorial | |
| 33 | - Allow: /discover | |
| 34 | - Allow: /stories | |
| 35 | - Allow: /topic | |
| 36 | - | |
| 37 | - # Platform: Product & Marketing | |
| 38 | - Allow: /pro | |
| 39 | - Allow: /download | |
| 40 | - Allow: /jobs | |
| 41 | - Allow: /go | |
| 42 | - Allow: /getstarted | |
| 43 | - | |
| 44 | - # Platform: Corporate | |
| 45 | - Allow: /company | |
| 46 | - | |
| 47 | - # Platform: Technical | |
| 48 | - Allow: /sitemap | |
| 49 | - Allow: /sitemapIndex | |
| 50 | - | |
| 51 | - # Block everything else (catches all UGC at root paths) | |
| 52 | 7 | Disallow: / |
| 53 | 8 | |
| 54 | - # Search engines and all other crawlers: index UGC, block low-value paths | |
| 55 | - User-Agent: * | |
| 56 | - Disallow: /search | |
| 57 | - Disallow: /you/ | |
| 58 | - Disallow: /stream | |
| 59 | - Disallow: /upload | |
| 60 | - Disallow: /settings | |
| 61 | - Disallow: /messages | |
| 62 | - Disallow: /*? | |
| 9 | + User-Agent: ClaudeBot | |
| 10 | + Disallow: / | |
| 63 | 11 | |
| 64 | - Sitemap: https://soundcloud.com/sitemap.xml | |
| 65 | - Sitemap: https://soundcloud.com/sitemapIndex.xml | |
| 12 | + User-Agent: CCBot | |
| 13 | + Disallow: / | |
| 14 | + | |
| 15 | + User-Agent: GPTBot | |
| 16 | + Disallow: / | |
| 17 | + | |
| 18 | + User-Agent: Google-Extended | |
| 19 | + Disallow: / | |
| 20 | + | |
| 21 | + User-Agent: Meta-ExternalAgent | |
| 22 | + Disallow: / | |
| 23 | + | |
| 24 | + User-Agent: omgili | |
| 25 | + Disallow: / | |
| 26 | + | |
| 27 | + User-Agent: omgilibot | |
| 28 | + Disallow: / | |
| 29 | + | |
| 30 | + User-Agent: Brightbot | |
| 31 | + Disallow: / | |
| 32 | + | |
| 33 | + sitemap: https://soundcloud.com/sitemap.xml | |
| 34 | + sitemap: https://soundcloud.com/sitemapIndex.xml |