Diffbot
Diffbot · Structured data extraction
65 of 1195 monitored sites (5.4%) block Diffbot by name.
Diffbot blocking over time share of monitored sites · weekly snapshot
Sites blocking
Diffbot
2026-06-23
2026-08-04
19 earlier weeks omitted (archive building) · 7 weekly snapshots with reliable coverage
Recent Diffbot events
- 2026-08-04 — Commonwealth Bank: Added rules for AI crawler Diffbot (Diffbot) diff
- 2026-07-29 — Diffbot: Added sitemap https://www.diffbot.com/sitemap.xml diff
- 2026-07-01 — Volvo Cars: Added Disallow: /*/build/* for Diffbot; Added Disallow: /*/order-request/* for Diffbot; Removed Disallow: /*/build/*?* for Diffbot … and 1 more change(s) diff
- 2026-06-30 — Indeed: Added rules for AI crawler Diffbot (Diffbot) diff
- 2026-06-29 — Kimberly-Clark: Added rules for AI crawler Diffbot (Diffbot) diff
- 2026-06-29 — Corriere della Sera: Blocked user-agent ‘Diffbot-User’ entirely diff
- 2026-06-28 — Corriere della Sera: Removed all rules for user-agent ‘Diffbot-User’ diff
- 2026-06-28 — Corriere della Sera: Blocked user-agent ‘Diffbot-User’ entirely diff
- 2026-06-27 — Corriere della Sera: Removed all rules for user-agent ‘Diffbot-User’ diff
- 2026-06-27 — Corriere della Sera: Blocked user-agent ‘Diffbot-User’ entirely diff
- 2026-06-27 — Corriere della Sera: Removed all rules for user-agent ‘Diffbot-User’ diff
- 2026-06-26 — Corriere della Sera: Blocked user-agent ‘Diffbot-User’ entirely diff
- 2026-06-17 — Regeneron Pharmaceuticals: Added rules for AI crawler Diffbot (Diffbot) diff
- 2026-06-17 — Evonik: Added rules for AI crawler Diffbot (Diffbot) diff
- 2026-06-17 — Nikkei: Blocked Diffbot (Diffbot) entirely diff
Sites blocking Diffbot
| Site | Sector | Badge |
|---|---|---|
| Amazon | Retailing |  |
| Ars Technica | Technology |  |
| Axios | News |  |
| Bandcamp | Music |  |
| BBC | Broadcast News |  |
| Behance | Graphics Multimedia and Web Design |  |
| Bild | News |  |
| Bloomberg | News & Financial |  |
| Business Insider | Business |  |
| BuzzFeed | Digital Media |  |
| Canva | Web |  |
| CNN | Broadcast News |  |
| Daily Mail | News |  |
| De Telegraaf | News |  |
| Der Spiegel | News |  |
| Die Zeit | News |  |
| Financial Times | News & Financial |  |
| Forbes | Business |  |
| Genius | Music |  |
| Halliburton | Energy |  |
| HuffPost | Digital Media |  |
| IGN | Web |  |
| Le Figaro | News |  |
| Le Monde | News |  |
| Letterboxd | Arts and Entertainment |  |
| Professional Network |  | |
| Los Angeles Times | News |  |
| Midjourney | LLM |  |
| Msn | Web |  |
| NBC News | Broadcast News |  |
| Newsweek | Magazine |  |
| Nikkei | News & Financial |  |
| Nintendo | Video Games |  |
| NPR | Public Radio |  |
| Pixiv | Art |  |
| Politico | News |  |
| Proofpoint | Cybersecurity |  |
| Quora | Q&A |  |
| Scribd | Public Records and Directories |  |
| Sky News | Broadcast News |  |
| SlideShare | Public Records and Directories |  |
| Snapchat | Messaging |  |
| Storyblocks | Stock media |  |
| TechCrunch | Technology |  |
| Telstra | Telecommunications |  |
| The Asahi Shimbun | News |  |
| The Atlantic | Magazine |  |
| The Globe and Mail | News |  |
| The Hill | Politics |  |
| The New York Times | News |  |
| The New Yorker | Magazine |  |
| The Sydney Morning Herald | News |  |
| The Telegraph | News |  |
| The Verge | Technology |  |
| The Washington Post | News |  |
| Toronto Star | News |  |
| Toyota | Motor Vehicles |  |
| Uol | Web |  |
| USA Today | News |  |
| Vimeo | Video |  |
| Vox | News |  |
| Weather | Weather |  |
| wikiHow | Dictionaries and Encyclopedias |  |
| Wired | Technology |  |
| Yahoo Search | Search engine |  |