BBC / robots.txt
← back to bbc.com · change detected 29d ago (2026-10-08T16:07:06Z) · modified +6 −4
What changed
- 🤖 AIAdded Allow: /storyworks for OAI-SearchBot
- 🤖 AIAdded Allow: /storyworks/ for OAI-SearchBot
Diff old (2026-08-24T13:15:37Z) → new (2026-10-08T16:07:06Z)
| @@ -1,5 +1,5 @@ | ||
| 1 | 1 | |
| 2 | - # version: 958ed1a58c8ce418427283859a5538a5ce0eb62d | |
| 2 | + # version: 20b91abf30af3b3b076745396ba368978384f4d7 | |
| 3 | 3 | # The BBC's Terms of Use: https://www.bbc.co.uk/terms |
| 4 | 4 | # - Explain the rules for using our services |
| 5 | 5 | # - Tell you what you can do with our content |
| @@ -165,6 +165,11 @@ | ||
| 165 | 165 | Allow: /storyworks |
| 166 | 166 | Allow: /storyworks/ |
| 167 | 167 | |
| 168 | + User-agent: OAI-SearchBot | |
| 169 | + Disallow: / | |
| 170 | + Allow: /storyworks | |
| 171 | + Allow: /storyworks/ | |
| 172 | + | |
| 168 | 173 | User-Agent: PerplexityBot |
| 169 | 174 | Disallow: / |
| 170 | 175 | |
| @@ -175,9 +180,6 @@ | ||
| 175 | 180 | Disallow: / |
| 176 | 181 | |
| 177 | 182 | User-agent: meta-externalagent |
| 178 | - Disallow: / | |
| 179 | - | |
| 180 | - User-agent: OAI-SearchBot | |
| 181 | 183 | Disallow: / |
| 182 | 184 | |
| 183 | 185 | User-agent: YandexAdditional |