CCBot

Recent CCBot events

• 2026-08-04 — Commonwealth Bank: Added rules for AI crawler CCBot (Common Crawl) diff
• 2026-07-29 — Lithia Motors: Blocked CCBot (Common Crawl) entirely diff
• 2026-07-22 — Roblox: Unblocked CCBot (Common Crawl) - rules removed diff
• 2026-07-21 — SoundCloud: Removed Allow: /$ for CCBot; Removed Allow: /accessibility-statement for CCBot; Removed Allow: /community-guidelines for CCBot … and 11 more change(s) diff
• 2026-07-21 — Roblox: Blocked CCBot (Common Crawl) entirely diff
• 2026-07-21 — SoundCloud: Added Allow: /$ for CCBot; Added Allow: /accessibility-statement for CCBot; Added Allow: /community-guidelines for CCBot … and 11 more change(s) diff
• 2026-07-21 — Roblox: Unblocked CCBot (Common Crawl) - rules removed diff
• 2026-07-20 — Roblox: Blocked CCBot (Common Crawl) entirely diff
• 2026-07-11 — Hulu: Removed Disallow: /start for CCBot diff
• 2026-07-08 — Trend Micro: Removed Allow: *.css$ for CCBot; Removed Allow: *.js$ for CCBot; Removed Disallow: /*/business/parsing-pages/ for CCBot … and 5 more change(s) diff
• 2026-07-08 — Hulu: Added rules for AI crawler CCBot (Common Crawl) diff
• 2026-07-07 — SoundCloud: Removed Allow: /$ for CCBot; Removed Allow: /accessibility-statement for CCBot; Removed Allow: /community-guidelines for CCBot … and 11 more change(s) diff
• 2026-07-06 — SoundCloud: Added Allow: /$ for CCBot; Added Allow: /accessibility-statement for CCBot; Added Allow: /community-guidelines for CCBot … and 11 more change(s) diff
• 2026-07-01 — Volvo Cars: Added Disallow: /*/build/* for CCBot; Added Disallow: /*/order-request/* for CCBot; Removed Disallow: /*/build/*?* for CCBot … and 1 more change(s) diff
• 2026-06-27 — Grupo Bimbo: Added rules for AI crawler CCBot (Common Crawl) diff
• 2026-06-26 — SoundCloud: Removed Allow: /$ for CCBot; Removed Allow: /accessibility-statement for CCBot; Removed Allow: /community-guidelines for CCBot … and 11 more change(s) diff
• 2026-06-25 — SoundCloud: Added Allow: /$ for CCBot; Added Allow: /accessibility-statement for CCBot; Added Allow: /community-guidelines for CCBot … and 11 more change(s) diff
• 2026-06-17 — Regeneron Pharmaceuticals: Added rules for AI crawler CCBot (Common Crawl) diff
• 2026-06-17 — Evonik: Added rules for AI crawler CCBot (Common Crawl) diff
• 2026-06-17 — Nikkei: Blocked CCBot (Common Crawl) entirely diff

Sites blocking CCBot

Site	Sector	Badge
ABC News	Broadcast News
Adobe Stock	Stock media
AI21 Labs	LLM
Amazon	Retailing
ArcelorMittal	Steel
Archive of Our Own	Books
Arrow Electronics	Technology
Ars Technica	Technology
Axios	News
Bandcamp	Music
Barracuda	Cybersecurity
Baxter International	Health Care
BBC	Broadcast News
Behance	Graphics Multimedia and Web Design
Bild	News
bioRxiv	Preprint server
Bloomberg	News & Financial
Burlington Stores	Retailing
Business Insider	Business
BuzzFeed	Digital Media
Canva	Web
ChatGPT	LLM
CNN	Broadcast News
Corriere della Sera	News
Costa Rica (Presidencia)	Americas
Daily Mail	News
De Telegraaf	News
Der Spiegel	News
Die Zeit	News
DOAJ	Open access directory
eBay	Technology
El País	News
EPFL	University
ESPN	Web
Expeditors International	Transportation
Figma Community	Graphics Multimedia and Web Design
Financial Times	News & Financial
Forbes	Business
Genius	Music
Globo	Web
Goodreads	Books
Guyana (Office of the President)	Americas
HuffPost	Digital Media
Iceland (Government.is)	Europe
IGN	Web
Insight Enterprises	Technology
Kimberly-Clark	Consumer Goods
Le Figaro	News
Le Monde	News
Letterboxd	Arts and Entertainment
LinkedIn	Professional Network
Lithia Motors	Retailing
Midjourney	LLM
Msn	Web
NBC News	Broadcast News
Newsweek	Magazine
Nikkei	News & Financial
Nintendo	Video Games
Northrop Grumman	Aerospace & Defense
NPR	Public Radio
Parker Hannifin	Industrials
Pixiv	Art
Politico	News
Pond5	Stock media
Prudential Financial	Financials
Scribd	Public Records and Directories
Sempra	Energy
ServiceNow	Technology
Shutterstock	Stock media
Sky News	Broadcast News
Slate	Digital Media
SlideShare	Public Records and Directories
Snapchat	Messaging
Sony Music	Music
SoundCloud	Music
Storyblocks	Stock media
TechCrunch	Technology
Telstra	Telecommunications
Tenet Healthcare	Health Care
The Asahi Shimbun	News
The Associated Press	Newswire
The Atlantic	Magazine
The Economist	Magazine
The Globe and Mail	News
The Guardian	News
The Hill	Politics
The Intercept	Investigative
The New York Times	News
The New Yorker	Magazine
The Sydney Morning Herald	News
The Telegraph	News
The Verge	Technology
The Washington Post	News
TikTok	Video
Toronto Star	News
Toyota	Motor Vehicles
Tripadvisor	Web
Tumblr	Social Network
Uol	Web
USA Today	News
Vimeo	Video
Vox	News
Weather	Weather
wikiHow	Dictionaries and Encyclopedias
Wired	Technology
WPP	Advertising
Yahoo Search	Search engine
Yelp	Web