Trends

According to an analysis by Elie Berreby: Across the web's 1,000 most-visited sites, 15.4% block an AI crawler, and 10.8% single out OpenAI.

Copy

≥1 AI 15.4%

OpenAI 10.8%

GPTBot 10.5%

Google-Ext 9%

According to an analysis by Elie Berreby: Corporate America is still largely open: 5.7% of the US 500 websites block at least one AI crawler; only 3.2% single out GPTBot.

Copy

≥1 AI 5.7%

OpenAI 3.4%

GPTBot 3.2%

Google-Ext 2.8%

According to an analysis by Elie Berreby: The tech-heavy Nasdaq-100 is split on AI scraping: 3.2% block OpenAI's crawlers; 1.1% opt out of Gemini training via Google-Extended.

Copy

≥1 AI 8.4%

OpenAI 3.2%

GPTBot 3.2%

Google-Ext 1.1%

According to an analysis by Elie Berreby: Even AI's own builders sometimes pull up the ladder: 11.9% of AI models block at least one crawler; only 7.1% block GPTBot.

Copy

≥1 AI 11.9%

OpenAI 7.1%

GPTBot 7.1%

Google-Ext 11.9%

According to an analysis by Elie Berreby: Only 7.3% of cybersecurity vendors block at least one AI crawler; none single out GPTBot.

Copy

≥1 AI 7.3%

OpenAI —

GPTBot —

Google-Ext —

According to an analysis by Elie Berreby: The crawlers that index the web write their own robots.txt rules: 1.9% block at least one AI crawler; 1.9% block OpenAI.

Copy

≥1 AI 1.9%

OpenAI 1.9%

GPTBot 1.9%

Google-Ext 1.9%

According to an analysis by Elie Berreby: Creative rights-holders are among the most defensive: 27% block an AI crawler, and each names GPTBot.

Copy

≥1 AI 27%

OpenAI 27%

GPTBot 27%

Google-Ext 16.2%

According to an analysis by Elie Berreby: The platforms whose content trains the models are pushing back: 43.6% of knowledge & UGC sites block an AI crawler; only 28.2% single out OpenAI.

Copy

≥1 AI 43.6%

OpenAI 28.2%

GPTBot 28.2%

Google-Ext 12.8%

According to an analysis by Elie Berreby: Scholarly archives are pushing back on AI scraping: 26.3% of academic & open-research sites block at least one AI crawler; only 21.1% single out GPTBot.

Copy

≥1 AI 26.3%

OpenAI 21.1%

GPTBot 21.1%

Google-Ext 15.8%

According to an analysis by Elie Berreby: University homepages stay mostly open: only 2.4% of top universities block at least one AI crawler; only 2.4% single out OpenAI.

Copy

≥1 AI 2.4%

OpenAI 2.4%

GPTBot 2.4%

Google-Ext —

According to an analysis by Elie Berreby: From the FTSE to the Nikkei, AI blocking is uneven: 6.5% of blue-chip index constituents block an AI crawler; only 4% single out OpenAI.

Copy

≥1 AI 6.5%

OpenAI 4%

GPTBot 4%

Google-Ext 3.2%

According to an analysis by Elie Berreby: Banks and fintechs guard customer data, but their websites are open to AI crawlers: only none block at least one AI crawler; none single out OpenAI.

Copy

≥1 AI —

OpenAI —

GPTBot —

Google-Ext —

According to an analysis by Elie Berreby: Retailers are slowly fencing their catalogs: 30% of major e-commerce sites block an AI crawler, and 25% single out OpenAI.

Copy

≥1 AI 30%

OpenAI 25%

GPTBot 25%

Google-Ext 15%

According to an analysis by Elie Berreby: 2.6% of major European companies block OpenAI's crawlers. Robots.txt is quietly becoming an AI opt-out form.

Copy

≥1 AI 4.5%

OpenAI 2.6%

GPTBot 2.6%

Google-Ext 1.9%

According to an analysis by Elie Berreby: The platforms sitting on the most user data are the most guarded: 56% of major social networks block an AI crawler; only 48% single out GPTBot.

Copy

≥1 AI 56%

OpenAI 52%

GPTBot 48%

Google-Ext 40%

According to an analysis by Elie Berreby: Government websites stay wide open: only 7.5% of national governments restrict at least one AI crawler.

Copy

≥1 AI 7.5%

OpenAI 7.5%

GPTBot 7.5%

Google-Ext 7.5%

According to an analysis by Elie Berreby: Publishers protect their content: 53% block GPTBot, and 77.3% block at least one AI crawler.

Copy

≥1 AI 77.3%

OpenAI 56.1%

GPTBot 53%

Google-Ext 50%