NeuralCrawl

BBC / robots.txt snapshot

← back to bbc.com · fetched 2026-11-02T12:53:58Z (5d ago) · HTTP 200 · 6250 bytes · sha256 c2bb93901c38a3de · raw

final URL: https://www.bbc.com/robots.txt

1
2# version: 2323e66fcb22a9568be0b7ab535fafcd2bf666ad
3# The BBC's Terms of Use: https://www.bbc.co.uk/terms
4# - Explain the rules for using our services
5# - Tell you what you can do with our content
6#
7# In short: Please use our site like a human, not a robot.
8# That means:
9# - No scraping, crawling, or systematic extraction of content
10# - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)
11# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content
12# - No creating datasets from BBC content
13# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market
14# - No using BBC content to create summaries for your own use
15# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)
16# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law
17
18# TL;DR: Browse, read, watch, enjoy - like a human.
19#
20
21# HTTPS www.bbc.com
22
23User-agent: *
24Sitemap: https://www.bbc.com/afrique/sitemap.xml
25Sitemap: https://www.bbc.com/arabic/sitemap.xml
26Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml
27Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml
28Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml
29Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml
30Sitemap: https://www.bbc.com/bengali/sitemap.xml
31Sitemap: https://www.bbc.com/burmese/sitemap.xml
32Sitemap: https://www.bbc.com/gahuza/sitemap.xml
33Sitemap: https://www.bbc.com/hausa/sitemap.xml
34Sitemap: https://www.bbc.com/hindi/sitemap.xml
35Sitemap: https://www.bbc.com/indonesia/sitemap.xml
36Sitemap: https://www.bbc.com/mundo/sitemap.xml
37Sitemap: https://www.bbc.com/pashto/sitemap.xml
38Sitemap: https://www.bbc.com/persian/sitemap.xml
39Sitemap: https://www.bbc.com/portuguese/sitemap.xml
40Sitemap: https://www.bbc.com/russian/sitemap.xml
41Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml
42Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml
43Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive_video.xml
44Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive.xml
45Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-news.xml
46Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-video.xml
47Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml
48Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml
49Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
50Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
51Sitemap: https://www.bbc.com/sitemaps/world-service-homepages.xml
52Sitemap: https://www.bbc.com/sport/sitemap.xml
53Sitemap: https://www.bbc.com/swahili/sitemap.xml
54Sitemap: https://www.bbc.com/tajik/sitemap.xml
55Sitemap: https://www.bbc.com/turkce/sitemap.xml
56Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml
57Sitemap: https://www.bbc.com/ukrainian/sitemap.xml
58Sitemap: https://www.bbc.com/urdu/sitemap.xml
59Sitemap: https://www.bbc.com/uzbek/sitemap.xml
60Sitemap: https://www.bbc.com/vietnamese/sitemap.xml
61Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml
62Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml
63
64Disallow: /asset/
65Disallow: /backstage/bbc-login-help/
66Disallow: /backstage/bbc-login-help$
67Disallow: /bitesize/search$
68Disallow: /bitesize/search/
69Disallow: /bitesize/search?
70Disallow: /cbbc/search/
71Disallow: /cbbc/search$
72Disallow: /cbbc/search?
73Disallow: /cbeebies/search/
74Disallow: /cbeebies/search$
75Disallow: /cbeebies/search?
76Disallow: /chwilio/
77Disallow: /chwilio$
78Disallow: /chwilio?
79Disallow: /education/blocks$
80Disallow: /education/blocks/
81Disallow: /newsround
82Disallow: /search/
83Disallow: /search$
84Disallow: /search?
85Disallow: /food/favourites
86Disallow: /food/search*?*
87Disallow: /food/recipes/search*?*
88Disallow: /education/my$
89Disallow: /education/my/
90Disallow: /bitesize/my$
91Disallow: /bitesize/my/
92Disallow: /food/recipes/*/shopping-list
93Disallow: /food/menus/*/shopping-list
94Disallow: /news/0
95Disallow: /sport/alpha/
96Disallow: /ugc$
97Disallow: /ugc/
98Disallow: /ugcsupport$
99Disallow: /ugcsupport/
100Disallow: /userinfo/
101Disallow: /userinfo
102Disallow: /u5llnop$
103Disallow: /u5llnop/
104Disallow: /sounds/search$
105Disallow: /sounds/search/
106Disallow: /sounds/search?
107Disallow: /ws/includes
108Disallow: /radio/imda
109Disallow: /storyworks/preview/*
110Disallow: /rd/search$
111Disallow: /rd/search/
112Disallow: /rd/search?
113
114User-agent: Amazonbot
115Disallow: /
116
117User-agent: magpie-crawler
118Disallow: /
119
120User-agent: CCBot
121Disallow: /
122
123User-Agent: omgili
124Disallow: /
125
126User-Agent: omgilibot
127Disallow: /
128
129User-agent: Claude-Web
130Disallow: /
131
132User-agent: ClaudeBot
133Disallow: /
134
135User-agent: anthropic-ai
136Disallow: /
137
138User-agent: cohere-ai
139Disallow: /
140
141User-agent: Bytespider
142Disallow: /
143
144User-agent: PetalBot
145Disallow: /
146
147User-agent: Scrapy
148Disallow: /
149
150User-agent: Applebot-Extended
151Disallow: /
152
153User-agent: GPTBot
154Disallow: /
155Allow: /storyworks
156Allow: /storyworks/
157
158User-agent: ChatGPT-User
159Disallow: /
160Allow: /storyworks
161Allow: /storyworks/
162
163User-agent: Google-Extended
164Disallow: /
165Allow: /storyworks
166Allow: /storyworks/
167
168User-agent: OAI-SearchBot
169Disallow: /
170Allow: /storyworks
171Allow: /storyworks/
172
173User-Agent: PerplexityBot
174Disallow: /
175
176User-agent: Perplexity-User
177Disallow: /
178
179User-agent: Google-CloudVertexBot
180Disallow: /
181
182User-agent: meta-externalagent
183Disallow: /
184
185User-agent: YandexAdditional
186Disallow: /
187
188User-agent: YandexAdditionalBot
189Disallow: /
190
191User-agent: TurnitinBot
192Disallow: /
193
194User-agent: Brightbot
195Disallow: /
196
197User-agent: ApifyBot
198Disallow: /
199
200User-agent: ApifyWebsiteContentCrawler
201Disallow: /
202
203User-agent: Diffbot
204Disallow: /
205
206User-agent: Diffbot-User
207Disallow: /
208
209User-agent: ExaBot
210Disallow: /
211
212User-agent: TavilyBot
213Disallow: /
214
215User-agent: ShapBot
216Disallow: /
217
218User-agent: YouBot
219Disallow: /
220
221User-agent: FirecrawlAgent
222Disallow: /
223
224User-agent: Amzn-SearchBot
225Disallow: /
226
227User-agent: Amzn-User
228Disallow: /
229
230User-agent: ProRataInc
231Disallow: /
232
233User-agent: CloudflareBrowserRenderingCrawler
234Disallow: /
235
236User-agent: AhrefsBot
237Disallow: /
238
239User-agent: Claude-SearchBot
240Disallow: /
241
242User-agent: DuckAssistBot
243Disallow: /