NeuralCrawl

DBLP / robots.txt

← back to dblp.org · change detected 5d ago (2026-09-09T18:36:35Z) · modified +71 −1

What changed

Diff old (2026-09-07T11:45:01Z) → new (2026-09-09T18:36:35Z)

@@ -1,68 +1,121 @@
11  User-agent: AddSearchBot
2+ User-agent: AgentTimes
23  User-agent: AI2Bot
4+ User-agent: AI2Bot-DeepResearchEval
35  User-agent: Ai2Bot-Dolma
46  User-agent: aiHitBot
7+ User-agent: AIWebIndex
8+ User-agent: amazon-kendra
9+ User-agent: amazon-QBusiness
510  User-agent: Amazonbot
11+ User-agent: AmazonBuyForMe
12+ User-agent: Amzn-SearchBot
13+ User-agent: Amzn-User
614  User-agent: Andibot
15+ User-agent: Anomura
716  User-agent: anthropic-ai
17+ User-agent: ApifyBot
18+ User-agent: ApifyWebsiteContentCrawler
819  User-agent: Applebot
920  User-agent: Applebot-Extended
21+ User-agent: Aranet-SearchBot
22+ User-agent: atlassian-bot
1023  User-agent: Awario
24+ User-agent: AzureAI-SearchBot
1125  User-agent: bedrockbot
1226  User-agent: bigsur.ai
27+ User-agent: Bravebot
28+ User-agent: Brightbot
1329  User-agent: Brightbot 1.0
30+ User-agent: BuddyBot
1431  User-agent: Bytespider
15- User-agent: CCBot
32+ User-agent: Channel3Bot
33+ User-agent: ChatGLM-Spider
1634  User-agent: ChatGPT Agent
1735  User-agent: ChatGPT-User
36+ User-agent: Claude-Code
1837  User-agent: Claude-SearchBot
1938  User-agent: Claude-User
2039  User-agent: Claude-Web
2140  User-agent: ClaudeBot
41+ User-agent: Cloudflare-AutoRAG
2242  User-agent: CloudVertexBot
43+ User-agent: Code
2344  User-agent: cohere-ai
2445  User-agent: cohere-training-data-crawler
2546  User-agent: Cotoyogi
47+ User-agent: CragCrawler
48+ User-agent: Crawl4AI
2649  User-agent: Crawlspace
50+ User-agent: Cursor
2751  User-agent: Datenbank Crawler
52+ User-agent: DeepSeekBot
2853  User-agent: Devin
2954  User-agent: Diffbot
3055  User-agent: DuckAssistBot
3156  User-agent: Echobot Bot
3257  User-agent: EchoboxBot
58+ User-agent: ExaBot
3359  User-agent: FacebookBot
3460  User-agent: facebookexternalhit
3561  User-agent: Factset_spyderbot
3662  User-agent: FirecrawlAgent
3763  User-agent: FriendlyCrawler
64+ User-agent: GeistHaus-PageFetcher
3865  User-agent: Gemini-Deep-Research
66+ User-agent: Google-Agent
3967  User-agent: Google-CloudVertexBot
4068  User-agent: Google-Extended
69+ User-agent: Google-Firebase
70+ User-agent: Google-Gemini-CLI
71+ User-agent: Google-NotebookLM
4172  User-agent: GoogleAgent-Mariner
73+ User-agent: GoogleAgent-URLContext
4274  User-agent: GoogleOther
4375  User-agent: GoogleOther-Image
4476  User-agent: GoogleOther-Video
4577  User-agent: GPTBot
78+ User-agent: HenkBot
79+ User-agent: iAskBot
80+ User-agent: iaskspider
4681  User-agent: iaskspider/2.0
82+ User-agent: IbouBot
4783  User-agent: ICC-Crawler
4884  User-agent: ImagesiftBot
85+ User-agent: imageSpider
4986  User-agent: img2dataset
5087  User-agent: ISSCyberRiskCrawler
88+ User-agent: kagi-fetcher
5189  User-agent: Kangaroo Bot
90+ User-agent: Kimi-User
91+ User-agent: KlaviyoAIBot
92+ User-agent: KunatoCrawler
93+ User-agent: laion-huggingface-processor
94+ User-agent: LAIONDownloader
95+ User-agent: LCC
5296  User-agent: LinerBot
97+ User-agent: Linguee Bot
98+ User-agent: LinkupBot
99+ User-agent: Manus-User
53100  User-agent: meta-externalagent
54101  User-agent: Meta-ExternalAgent
55102  User-agent: meta-externalfetcher
56103  User-agent: Meta-ExternalFetcher
104+ User-agent: meta-webindexer
57105  User-agent: MistralAI-User
58106  User-agent: MistralAI-User/1.0
107+ User-agent: Mozilla-Tabstack
59108  User-agent: MyCentralAIScraperBot
109+ User-agent: NagetBot
60110  User-agent: netEstate Imprint Crawler
111+ User-agent: newsai
112+ User-agent: NotebookLM
61113  User-agent: NovaAct
62114  User-agent: OAI-SearchBot
63115  User-agent: omgili
64116  User-agent: omgilibot
65117  User-agent: OpenAI
118+ User-agent: opencode
66119  User-agent: Operator
67120  User-agent: PanguBot
68121  User-agent: Panscient
@@ -71,26 +124,43 @@
71124  User-agent: PerplexityBot
72125  User-agent: PetalBot
73126  User-agent: PhindBot
127+ User-agent: Poggio-Citations
74128  User-agent: Poseidon Research Crawler
75129  User-agent: QualifiedBot
130+ User-agent: Querit-SearchBot
131+ User-agent: QueritBot
76132  User-agent: QuillBot
77133  User-agent: quillbot.com
78134  User-agent: SBIntuitionsBot
79135  User-agent: Scrapy
80136  User-agent: SemrushBot-OCOB
81137  User-agent: SemrushBot-SWA
138+ User-agent: Shap-User
139+ User-agent: ShapBot
82140  User-agent: Sidetrade indexer bot
141+ User-agent: Spider
142+ User-agent: TavilyBot
143+ User-agent: Terra Cotta
144+ User-agent: TerraCotta
83145  User-agent: Thinkbot
84146  User-agent: TikTokSpider
85147  User-agent: Timpibot
148+ User-agent: TongyiBot
149+ User-agent: Trae
150+ User-agent: TwinAgent
151+ User-agent: UseAI
86152  User-agent: VelenPublicWebCrawler
87153  User-agent: WARDBot
88154  User-agent: Webzio-Extended
155+ User-agent: webzio-extended
89156  User-agent: wpbot
157+ User-agent: WRTNBot
90158  User-agent: YaK
91159  User-agent: YandexAdditional
92160  User-agent: YandexAdditionalBot
161+ User-agent: YiyanBot
93162  User-agent: YouBot
163+ User-agent: ZanistaBot
94164  Disallow: /
95165 
96166  User-agent: *