Trends

According to an analysis by Elie Berreby: Across the web's 1,000 most-visited sites, 15.1% block an AI crawler, and 10.7% single out OpenAI.

Copy

≥1 AI 15.1%

OpenAI 10.7%

GPTBot 10.5%

Google-Ext 9%

According to an analysis by Elie Berreby: Corporate America is still largely open: 6% of the US 500 websites block at least one AI crawler; only 3.2% single out GPTBot.

Copy

≥1 AI 6%

OpenAI 3.2%

GPTBot 3.2%

Google-Ext 2.6%

According to an analysis by Elie Berreby: The tech-heavy Nasdaq-100 is split on AI scraping: 2.1% block OpenAI's crawlers; 1.1% opt out of Gemini training via Google-Extended.

Copy

≥1 AI 7.4%

OpenAI 2.1%

GPTBot 2.1%

Google-Ext 1.1%

According to an analysis by Elie Berreby: Even AI's own builders sometimes pull up the ladder: 14.6% of AI models block at least one crawler; only 9.8% block GPTBot.

Copy

≥1 AI 14.6%

OpenAI 9.8%

GPTBot 9.8%

Google-Ext 14.6%

According to an analysis by Elie Berreby: Only 5.5% of cybersecurity vendors block at least one AI crawler; none single out GPTBot.

Copy

≥1 AI 5.5%

OpenAI —

GPTBot —

Google-Ext —

According to an analysis by Elie Berreby: The crawlers that index the web write their own robots.txt rules: 1.9% block at least one AI crawler; 1.9% block OpenAI.

Copy

≥1 AI 1.9%

OpenAI 1.9%

GPTBot 1.9%

Google-Ext 1.9%

According to an analysis by Elie Berreby: Creative rights-holders are among the most defensive: 25% block an AI crawler, and each names GPTBot.

Copy

≥1 AI 25%

OpenAI 25%

GPTBot 25%

Google-Ext 13.9%

According to an analysis by Elie Berreby: The platforms whose content trains the models are pushing back: 41% of knowledge & UGC sites block an AI crawler; only 28.2% single out OpenAI.

Copy

≥1 AI 41%

OpenAI 28.2%

GPTBot 28.2%

Google-Ext 12.8%

According to an analysis by Elie Berreby: Scholarly archives are pushing back on AI scraping: 15.8% of academic & open-research sites block at least one AI crawler; only 15.8% single out GPTBot.

Copy

≥1 AI 15.8%

OpenAI 15.8%

GPTBot 15.8%

Google-Ext 10.5%

According to an analysis by Elie Berreby: University homepages stay mostly open: only 4.9% of top universities block at least one AI crawler; only 4.9% single out OpenAI.

Copy

≥1 AI 4.9%

OpenAI 4.9%

GPTBot 4.9%

Google-Ext 2.4%

According to an analysis by Elie Berreby: From the FTSE to the Nikkei, AI blocking is uneven: 4.9% of blue-chip index constituents block an AI crawler; only 3.3% single out OpenAI.

Copy

≥1 AI 4.9%

OpenAI 3.3%

GPTBot 3.3%

Google-Ext 2.5%

According to an analysis by Elie Berreby: Banks and fintechs guard customer data, but their websites are open to AI crawlers: only none block at least one AI crawler; none single out OpenAI.

Copy

≥1 AI —

OpenAI —

GPTBot —

Google-Ext —

According to an analysis by Elie Berreby: Retailers are slowly fencing their catalogs: 25% of major e-commerce sites block an AI crawler, and 20% single out OpenAI.

Copy

≥1 AI 25%

OpenAI 20%

GPTBot 20%

Google-Ext 10%

According to an analysis by Elie Berreby: European corporate giants remain mostly welcoming to AI crawlers: 1.9% block OpenAI, and 3.9% block at least one crawler.

Copy

≥1 AI 3.9%

OpenAI 1.9%

GPTBot 1.9%

Google-Ext 1.3%

According to an analysis by Elie Berreby: The platforms sitting on the most user data are the most guarded: 52% of major social networks block an AI crawler; only 44% single out GPTBot.

Copy

≥1 AI 52%

OpenAI 48%

GPTBot 44%

Google-Ext 32%

According to an analysis by Elie Berreby: Government websites stay wide open: only 7.5% of national governments restrict at least one AI crawler.

Copy

≥1 AI 7.5%

OpenAI 7.5%

GPTBot 7.5%

Google-Ext 7.5%

According to an analysis by Elie Berreby: Publishers protect their content: 53% block GPTBot, and 75.8% block at least one AI crawler.

Copy

≥1 AI 75.8%

OpenAI 56.1%

GPTBot 53%

Google-Ext 53%