CCBot
Common Crawl · Open web corpus (used for LLM training)
110 of 1203 monitored sites (9.1%) block CCBot by name, 1 restrict it.
CCBot blocking over time share of monitored sites · weekly snapshot
Sites blocking
CCBot
2026-06-20
2026-11-07
5 earlier weeks omitted (archive building) · 21 weekly snapshots with reliable coverage
Recent CCBot events
- 2026-11-06 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-11-06 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-11-05 — Delta Air Lines: Removed rules for AI crawler CCBot (Common Crawl) diff
- 2026-11-05 — First American Financial: Blocked CCBot (Common Crawl) entirely diff
- 2026-11-05 — Best Buy: Removed Disallow: /[1.1] for CCBot diff
- 2026-11-04 — Bayer: Blocked CCBot (Common Crawl) entirely diff
- 2026-11-03 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-11-03 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-11-01 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-11-01 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-10-31 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-31 — Burlington Stores: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-10-30 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-10-30 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-30 — Pixiv: Added Allow: /info.php for CCBot diff
- 2026-10-30 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-10-29 — Delta Air Lines: Added rules for AI crawler CCBot (Common Crawl) diff
- 2026-10-28 — Heineken: Added rules for AI crawler CCBot (Common Crawl) diff
- 2026-10-27 — Mastodon: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-24 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-24 — Guyana (Office of the President): Blocked CCBot (Common Crawl) entirely diff
- 2026-10-24 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-10-23 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-23 — Advance Auto Parts: Unblocked CCBot (Common Crawl) - rules removed diff
- 2026-10-22 — Costa Rica (Presidencia): Blocked CCBot (Common Crawl) entirely diff
- 2026-10-22 — Sony Music: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-20 — Daily Mail: Added Allow: /llms.txt for CCBot diff
- 2026-10-19 — Iceland (Government.is): Blocked CCBot (Common Crawl) entirely diff
- 2026-10-18 — Advance Auto Parts: Blocked CCBot (Common Crawl) entirely diff
- 2026-10-18 — Expeditors International: Blocked CCBot (Common Crawl) entirely diff
Sites blocking CCBot
| Site | Sector | Badge |
|---|---|---|
| ABC News | Broadcast News |  |
| Adobe Stock | Stock media |  |
| Advance Auto Parts | Retailing |  |
| AliExpress | E-commerce |  |
| Amazon | Retailing |  |
| ArcelorMittal | Steel |  |
| Archive of Our Own | Books |  |
| Arrow Electronics | Technology |  |
| Ars Technica | Technology |  |
| Axios | News |  |
| Bandcamp | Music |  |
| Bayer | Pharmaceuticals |  |
| BBC | Broadcast News |  |
| Behance | Graphics Multimedia and Web Design |  |
| Bild | News |  |
| bioRxiv | Preprint server |  |
| Bloomberg | News & Financial |  |
| Business Insider | Business |  |
| BuzzFeed | Digital Media |  |
| Canva | Web |  |
| Casey's General Stores | Retailing |  |
| ChatGPT | LLM |  |
| China Mobile | Telecommunications |  |
| CNN | Broadcast News |  |
| Corriere della Sera | News |  |
| Costa Rica (Presidencia) | Americas |  |
| Daily Mail | News |  |
| De Telegraaf | News |  |
| Der Spiegel | News |  |
| Die Zeit | News |  |
| eBay | Technology |  |
| El País | News |  |
| ESPN | Web |  |
| Europe PMC | Research repository |  |
| Expeditors International | Transportation |  |
| Figma Community | Graphics Multimedia and Web Design |  |
| Financial Times | News & Financial |  |
| First American Financial | Financials |  |
| Forbes | Business |  |
| Genius | Music |  |
| Globo | Web |  |
| Goodreads | Books |  |
| Guyana (Office of the President) | Americas |  |
| HuffPost | Digital Media |  |
| Iceland (Government.is) | Europe |  |
| IGN | Web |  |
| Insight Enterprises | Technology |  |
| KeyCorp | Financials |  |
| Kimberly-Clark | Consumer Goods |  |
| Le Figaro | News |  |
| Le Monde | News |  |
| Letterboxd | Arts and Entertainment |  |
| Professional Network |  | |
| Lithia Motors | Retailing |  |
| Mastodon | Social Network (Fediverse) |  |
| Midjourney | LLM |  |
| Msn | Web |  |
| NBC News | Broadcast News |  |
| Newsweek | Magazine |  |
| Nikkei | News & Financial |  |
| Nintendo | Video Games |  |
| Northrop Grumman | Aerospace & Defense |  |
| NPR | Public Radio |  |
| Parker Hannifin | Industrials |  |
| Pixiv | Art |  |
| Politico | News |  |
| Pond5 | Stock media |  |
| Prudential Financial | Financials |  |
| Reliance Steel & Aluminum | Materials |  |
| Scribd | Public Records and Directories |  |
| ServiceNow | Technology |  |
| Shutterstock | Stock media |  |
| Sky News | Broadcast News |  |
| Slate | Digital Media |  |
| SlideShare | Public Records and Directories |  |
| Snapchat | Messaging |  |
| Sony Music | Music |  |
| SoundCloud | Music |  |
| Storyblocks | Stock media |  |
| TechCrunch | Technology |  |
| Telstra | Telecommunications |  |
| The Asahi Shimbun | News |  |
| The Associated Press | Newswire |  |
| The Atlantic | Magazine |  |
| The Economist | Magazine |  |
| The Globe and Mail | News |  |
| The Guardian | News |  |
| The Hill | Politics |  |
| The Intercept | Investigative |  |
| The New York Times | News |  |
| The New Yorker | Magazine |  |
| The Sydney Morning Herald | News |  |
| The Telegraph | News |  |
| The Verge | Technology |  |
| The Washington Post | News |  |
| Tidal | Music |  |
| TikTok | Video |  |
| Toronto Star | News |  |
| Tripadvisor | Web |  |
| Tumblr | Social Network |  |
| Uol | Web |  |
| USA Today | News |  |
| Vimeo | Video |  |
| Vox | News |  |
| Weather | Weather |  |
| wikiHow | Dictionaries and Encyclopedias |  |
| Wired | Technology |  |
| WPP | Advertising |  |
| Yahoo Search | Search engine |  |
| Yelp | Web |  |