NeuralCrawl

BBC / robots.txt snapshot

← back to bbc.com · fetched 2026-08-24T13:15:37Z (2mo ago) · HTTP 200 · 6130 bytes · sha256 34a6212c178b5845 · raw

final URL: https://www.bbc.com/robots.txt

1
2# version: 958ed1a58c8ce418427283859a5538a5ce0eb62d
3# The BBC's Terms of Use: https://www.bbc.co.uk/terms
4# - Explain the rules for using our services
5# - Tell you what you can do with our content
6#
7# In short: Please use our site like a human, not a robot.
8# That means:
9# - No scraping, crawling, or systematic extraction of content
10# - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)
11# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content
12# - No creating datasets from BBC content
13# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market
14# - No using BBC content to create summaries for your own use
15# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)
16# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law
17
18# TL;DR: Browse, read, watch, enjoy - like a human.
19#
20
21# HTTPS www.bbc.com
22
23User-agent: *
24Sitemap: https://www.bbc.com/afrique/sitemap.xml
25Sitemap: https://www.bbc.com/arabic/sitemap.xml
26Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml
27Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml
28Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml
29Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml
30Sitemap: https://www.bbc.com/bengali/sitemap.xml
31Sitemap: https://www.bbc.com/burmese/sitemap.xml
32Sitemap: https://www.bbc.com/gahuza/sitemap.xml
33Sitemap: https://www.bbc.com/hausa/sitemap.xml
34Sitemap: https://www.bbc.com/hindi/sitemap.xml
35Sitemap: https://www.bbc.com/indonesia/sitemap.xml
36Sitemap: https://www.bbc.com/mundo/sitemap.xml
37Sitemap: https://www.bbc.com/pashto/sitemap.xml
38Sitemap: https://www.bbc.com/persian/sitemap.xml
39Sitemap: https://www.bbc.com/portuguese/sitemap.xml
40Sitemap: https://www.bbc.com/russian/sitemap.xml
41Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml
42Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml
43Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive_video.xml
44Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive.xml
45Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-news.xml
46Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-video.xml
47Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml
48Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml
49Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
50Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
51Sitemap: https://www.bbc.com/sitemaps/world-service-homepages.xml
52Sitemap: https://www.bbc.com/sport/sitemap.xml
53Sitemap: https://www.bbc.com/swahili/sitemap.xml
54Sitemap: https://www.bbc.com/tajik/sitemap.xml
55Sitemap: https://www.bbc.com/turkce/sitemap.xml
56Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml
57Sitemap: https://www.bbc.com/ukrainian/sitemap.xml
58Sitemap: https://www.bbc.com/urdu/sitemap.xml
59Sitemap: https://www.bbc.com/uzbek/sitemap.xml
60Sitemap: https://www.bbc.com/vietnamese/sitemap.xml
61Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml
62Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml
63
64Disallow: /asset/
65Disallow: /backstage/bbc-login-help/
66Disallow: /backstage/bbc-login-help$
67Disallow: /bitesize/search$
68Disallow: /bitesize/search/
69Disallow: /bitesize/search?
70Disallow: /cbbc/search/
71Disallow: /cbbc/search$
72Disallow: /cbbc/search?
73Disallow: /cbeebies/search/
74Disallow: /cbeebies/search$
75Disallow: /cbeebies/search?
76Disallow: /chwilio/
77Disallow: /chwilio$
78Disallow: /chwilio?
79Disallow: /education/blocks$
80Disallow: /education/blocks/
81Disallow: /newsround
82Disallow: /search/
83Disallow: /search$
84Disallow: /search?
85Disallow: /food/favourites
86Disallow: /food/search*?*
87Disallow: /food/recipes/search*?*
88Disallow: /education/my$
89Disallow: /education/my/
90Disallow: /bitesize/my$
91Disallow: /bitesize/my/
92Disallow: /food/recipes/*/shopping-list
93Disallow: /food/menus/*/shopping-list
94Disallow: /news/0
95Disallow: /sport/alpha/
96Disallow: /ugc$
97Disallow: /ugc/
98Disallow: /ugcsupport$
99Disallow: /ugcsupport/
100Disallow: /userinfo/
101Disallow: /userinfo
102Disallow: /u5llnop$
103Disallow: /u5llnop/
104Disallow: /sounds/search$
105Disallow: /sounds/search/
106Disallow: /sounds/search?
107Disallow: /ws/includes
108Disallow: /radio/imda
109Disallow: /storyworks/preview/*
110Disallow: /rd/search$
111Disallow: /rd/search/
112Disallow: /rd/search?
113
114User-agent: Amazonbot
115Disallow: /
116
117User-agent: magpie-crawler
118Disallow: /
119
120User-agent: CCBot
121Disallow: /
122
123User-Agent: omgili
124Disallow: /
125
126User-Agent: omgilibot
127Disallow: /
128
129User-agent: Claude-Web
130Disallow: /
131
132User-agent: ClaudeBot
133Disallow: /
134
135User-agent: anthropic-ai
136Disallow: /
137
138User-agent: cohere-ai
139Disallow: /
140
141User-agent: Bytespider
142Disallow: /
143
144User-agent: PetalBot
145Disallow: /
146
147User-agent: Scrapy
148Disallow: /
149
150User-agent: Applebot-Extended
151Disallow: /
152
153User-agent: GPTBot
154Disallow: /
155Allow: /storyworks
156Allow: /storyworks/
157
158User-agent: ChatGPT-User
159Disallow: /
160Allow: /storyworks
161Allow: /storyworks/
162
163User-agent: Google-Extended
164Disallow: /
165Allow: /storyworks
166Allow: /storyworks/
167
168User-Agent: PerplexityBot
169Disallow: /
170
171User-agent: Perplexity-User
172Disallow: /
173
174User-agent: Google-CloudVertexBot
175Disallow: /
176
177User-agent: meta-externalagent
178Disallow: /
179
180User-agent: OAI-SearchBot
181Disallow: /
182
183User-agent: YandexAdditional
184Disallow: /
185
186User-agent: YandexAdditionalBot
187Disallow: /
188
189User-agent: TurnitinBot
190Disallow: /
191
192User-agent: Brightbot
193Disallow: /
194
195User-agent: ApifyBot
196Disallow: /
197
198User-agent: ApifyWebsiteContentCrawler
199Disallow: /
200
201User-agent: Diffbot
202Disallow: /
203
204User-agent: Diffbot-User
205Disallow: /
206
207User-agent: ExaBot
208Disallow: /
209
210User-agent: TavilyBot
211Disallow: /
212
213User-agent: ShapBot
214Disallow: /
215
216User-agent: YouBot
217Disallow: /
218
219User-agent: FirecrawlAgent
220Disallow: /
221
222User-agent: Amzn-SearchBot
223Disallow: /
224
225User-agent: Amzn-User
226Disallow: /
227
228User-agent: ProRataInc
229Disallow: /
230
231User-agent: CloudflareBrowserRenderingCrawler
232Disallow: /
233
234User-agent: AhrefsBot
235Disallow: /