NeuralCrawl

BBC / robots.txt

← back to bbc.com · change detected 20d ago (2026-07-17T12:07:56Z) · modified +3 −2

What changed

Diff old (2026-06-30T06:03:14Z)new (2026-07-17T12:07:56Z)

@@ -1,5 +1,5 @@
11 
2- # version: 639936d0986d41192594c2578b90fa570430630d
2+ # version: df99009430a1a8f8979042c24e21e9fecd8b21cd
33  # The BBC's Terms of Use: https://www.bbc.co.uk/terms
44  # - Explain the rules for using our services
55  # - Tell you what you can do with our content
@@ -26,8 +26,9 @@
2626  Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml
2727  Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml
2828  Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
29+ Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
30+ Sitemap: https://www.bbc.com/sitemaps/world-service-homepages.xml
2931  Sitemap: https://www.bbc.com/sport/sitemap.xml
30- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml
3132  Sitemap: https://www.bbc.com/afrique/sitemap.xml
3233  Sitemap: https://www.bbc.com/arabic/sitemap.xml
3334  Sitemap: https://www.bbc.com/bengali/sitemap.xml