Trends

“Across the web's 1,000 most-visited sites, 13.2% block an AI crawler (and 9.3% name OpenAI).” Elie Berreby

≥1 AI 13.2%

OpenAI 9.3%

GPTBot 9.1%

Google-Ext 7.9%

“Corporate America is still largely open: 5.1% of the US 500 websites block at least one AI crawler (only 2.4% name GPTBot).” Elie Berreby

≥1 AI 5.1%

OpenAI 2.4%

GPTBot 2.4%

Google-Ext 2%

“The tech-heavy Nasdaq-100 is split on AI scraping: 2.2% block OpenAI's crawlers; 1.1% opt out of Gemini training via Google-Extended.” Elie Berreby

≥1 AI 7.5%

OpenAI 2.2%

GPTBot 2.2%

Google-Ext 1.1%

“Even AI's own builders sometimes pull up the ladder: 8.1% of AI labs block at least one crawler (only 5.4% block GPTBot).” Elie Berreby

≥1 AI 8.1%

OpenAI 5.4%

GPTBot 5.4%

Google-Ext 8.1%

“For all their talk of web threats, only 7.3% of cybersecurity vendors block at least one AI crawler (none name GPTBot).” Elie Berreby

≥1 AI 7.3%

OpenAI —

GPTBot —

Google-Ext —

“The crawlers that index the web write their own robots.txt rules: 1.9% block at least one AI crawler; 1.9% block OpenAI.” Elie Berreby

≥1 AI 1.9%

OpenAI 1.9%

GPTBot 1.9%

Google-Ext 1.9%

“Creative rights-holders are among the most defensive: 25.7% block an AI crawler, and each names GPTBot.” Elie Berreby

≥1 AI 25.7%

OpenAI 25.7%

GPTBot 25.7%

Google-Ext 14.3%

“The platforms whose content trains the models are pushing back: 41% of knowledge & UGC sites block an AI crawler (only 28.2% name OpenAI).” Elie Berreby

≥1 AI 41%

OpenAI 28.2%

GPTBot 28.2%

Google-Ext 12.8%

“From the FTSE to the Nikkei, AI blocking is uneven: 5% of blue-chip index constituents block an AI crawler (only 3.3% name OpenAI).” Elie Berreby

≥1 AI 5%

OpenAI 3.3%

GPTBot 3.3%

Google-Ext 2.5%

“Banks and fintechs guard customer data, but their websites are open to AI crawlers: only 2.2% block at least one AI crawler (none name OpenAI).” Elie Berreby

≥1 AI 2.2%

OpenAI —

GPTBot —

Google-Ext —

“Retailers are slowly fencing their catalogs: 25% of major e-commerce sites block an AI crawler (and 20% name OpenAI).” Elie Berreby

≥1 AI 25%

OpenAI 20%

GPTBot 20%

Google-Ext 10%

“European corporate giants remain mostly welcoming to AI crawlers (1.9% block OpenAI, 3.2% block at least one crawler).” Elie Berreby

≥1 AI 3.2%

OpenAI 1.9%

GPTBot 1.9%

Google-Ext 1.3%

“The platforms sitting on the most user data are the most guarded: 52% of major social networks block an AI crawler (only 48% name GPTBot).” Elie Berreby

≥1 AI 52%

OpenAI 48%

GPTBot 48%

Google-Ext 36%

“Government websites stay wide open: only 7.5% of national governments restrict at least one AI crawler.” Elie Berreby

≥1 AI 7.5%

OpenAI 7.5%

GPTBot 7.5%

Google-Ext 7.5%

“Publishers protect their content: 54.7% block GPTBot, and 78.1% block at least one AI crawler.” Elie Berreby

≥1 AI 78.1%

OpenAI 57.8%

GPTBot 54.7%

Google-Ext 56.2%