NeuralCrawl

BBC / robots.txt snapshot

← back to bbc.com · fetched 2026-07-29T16:38:54Z (7d ago) · HTTP 200 · 5839 bytes · sha256 73f84b1043a22719 · raw

final URL: https://www.bbc.com/robots.txt

1
2# version: a294511aea2df10f410588a60a9157e46ab994a2
3# The BBC's Terms of Use: https://www.bbc.co.uk/terms
4# - Explain the rules for using our services
5# - Tell you what you can do with our content
6#
7# In short: Please use our site like a human, not a robot.
8# That means:
9# - No scraping, crawling, or systematic extraction of content
10# - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)
11# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content
12# - No creating datasets from BBC content
13# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market
14# - No using BBC content to create summaries for your own use
15# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)
16# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law
17
18# TL;DR: Browse, read, watch, enjoy - like a human.
19#
20
21# HTTPS www.bbc.com
22
23User-agent: *
24Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml
25Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml
26Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml
27Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml
28Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
29Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
30Sitemap: https://www.bbc.com/sitemaps/world-service-homepages.xml
31Sitemap: https://www.bbc.com/sport/sitemap.xml
32Sitemap: https://www.bbc.com/afrique/sitemap.xml
33Sitemap: https://www.bbc.com/arabic/sitemap.xml
34Sitemap: https://www.bbc.com/bengali/sitemap.xml
35Sitemap: https://www.bbc.com/burmese/sitemap.xml
36Sitemap: https://www.bbc.com/gahuza/sitemap.xml
37Sitemap: https://www.bbc.com/hausa/sitemap.xml
38Sitemap: https://www.bbc.com/hindi/sitemap.xml
39Sitemap: https://www.bbc.com/indonesia/sitemap.xml
40Sitemap: https://www.bbc.com/mundo/sitemap.xml
41Sitemap: https://www.bbc.com/pashto/sitemap.xml
42Sitemap: https://www.bbc.com/persian/sitemap.xml
43Sitemap: https://www.bbc.com/portuguese/sitemap.xml
44Sitemap: https://www.bbc.com/russian/sitemap.xml
45Sitemap: https://www.bbc.com/swahili/sitemap.xml
46Sitemap: https://www.bbc.com/tajik/sitemap.xml
47Sitemap: https://www.bbc.com/turkce/sitemap.xml
48Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml
49Sitemap: https://www.bbc.com/ukrainian/sitemap.xml
50Sitemap: https://www.bbc.com/urdu/sitemap.xml
51Sitemap: https://www.bbc.com/uzbek/sitemap.xml
52Sitemap: https://www.bbc.com/vietnamese/sitemap.xml
53Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml
54Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml
55Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml
56Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml
57Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml
58Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml
59
60Disallow: /asset/
61Disallow: /backstage/bbc-login-help/
62Disallow: /backstage/bbc-login-help$
63Disallow: /bitesize/search$
64Disallow: /bitesize/search/
65Disallow: /bitesize/search?
66Disallow: /cbbc/search/
67Disallow: /cbbc/search$
68Disallow: /cbbc/search?
69Disallow: /cbeebies/search/
70Disallow: /cbeebies/search$
71Disallow: /cbeebies/search?
72Disallow: /chwilio/
73Disallow: /chwilio$
74Disallow: /chwilio?
75Disallow: /education/blocks$
76Disallow: /education/blocks/
77Disallow: /newsround
78Disallow: /search/
79Disallow: /search$
80Disallow: /search?
81Disallow: /food/favourites
82Disallow: /food/search*?*
83Disallow: /food/recipes/search*?*
84Disallow: /education/my$
85Disallow: /education/my/
86Disallow: /bitesize/my$
87Disallow: /bitesize/my/
88Disallow: /food/recipes/*/shopping-list
89Disallow: /food/menus/*/shopping-list
90Disallow: /news/0
91Disallow: /sport/alpha/
92Disallow: /ugc$
93Disallow: /ugc/
94Disallow: /ugcsupport$
95Disallow: /ugcsupport/
96Disallow: /userinfo/
97Disallow: /userinfo
98Disallow: /u5llnop$
99Disallow: /u5llnop/
100Disallow: /sounds/search$
101Disallow: /sounds/search/
102Disallow: /sounds/search?
103Disallow: /ws/includes
104Disallow: /radio/imda
105Disallow: /storyworks/preview/*
106Disallow: /rd/search$
107Disallow: /rd/search/
108Disallow: /rd/search?
109
110User-agent: Amazonbot
111Disallow: /
112
113User-agent: magpie-crawler
114Disallow: /
115
116User-agent: CCBot
117Disallow: /
118
119User-Agent: omgili
120Disallow: /
121
122User-Agent: omgilibot
123Disallow: /
124
125User-agent: Claude-Web
126Disallow: /
127
128User-agent: ClaudeBot
129Disallow: /
130
131User-agent: anthropic-ai
132Disallow: /
133
134User-agent: cohere-ai
135Disallow: /
136
137User-agent: Bytespider
138Disallow: /
139
140User-agent: PetalBot
141Disallow: /
142
143User-agent: Scrapy
144Disallow: /
145
146User-agent: Applebot-Extended
147Disallow: /
148
149User-agent: GPTBot
150Disallow: /
151Allow: /storyworks
152Allow: /storyworks/
153
154User-agent: ChatGPT-User
155Disallow: /
156Allow: /storyworks
157Allow: /storyworks/
158
159User-agent: Google-Extended
160Disallow: /
161Allow: /storyworks
162Allow: /storyworks/
163
164User-Agent: PerplexityBot
165Disallow: /
166
167User-agent: Perplexity-User
168Disallow: /
169
170User-agent: Google-CloudVertexBot
171Disallow: /
172
173User-agent: meta-externalagent
174Disallow: /
175
176User-agent: OAI-SearchBot
177Disallow: /
178
179User-agent: YandexAdditional
180Disallow: /
181
182User-agent: YandexAdditionalBot
183Disallow: /
184
185User-agent: TurnitinBot
186Disallow: /
187
188User-agent: Brightbot
189Disallow: /
190
191User-agent: ApifyBot
192Disallow: /
193
194User-agent: ApifyWebsiteContentCrawler
195Disallow: /
196
197User-agent: Diffbot
198Disallow: /
199
200User-agent: Diffbot-User
201Disallow: /
202
203User-agent: ExaBot
204Disallow: /
205
206User-agent: TavilyBot
207Disallow: /
208
209User-agent: ShapBot
210Disallow: /
211
212User-agent: YouBot
213Disallow: /
214
215User-agent: FirecrawlAgent
216Disallow: /
217
218User-agent: Amzn-SearchBot
219Disallow: /
220
221User-agent: Amzn-User
222Disallow: /
223
224User-agent: ProRataInc
225Disallow: /
226
227User-agent: CloudflareBrowserRenderingCrawler
228Disallow: /
229
230User-agent: AhrefsBot
231Disallow: /