NeuralCrawl

DBLP / robots.txt snapshot

← back to dblp.org · fetched 2026-09-09T18:36:35Z (5d ago) · HTTP 200 · 4125 bytes · sha256 e7c9d8ab803235ba · raw

final URL: https://dblp.org/robots.txt

1User-agent: AddSearchBot
2User-agent: AgentTimes
3User-agent: AI2Bot
4User-agent: AI2Bot-DeepResearchEval
5User-agent: Ai2Bot-Dolma
6User-agent: aiHitBot
7User-agent: AIWebIndex
8User-agent: amazon-kendra
9User-agent: amazon-QBusiness
10User-agent: Amazonbot
11User-agent: AmazonBuyForMe
12User-agent: Amzn-SearchBot
13User-agent: Amzn-User
14User-agent: Andibot
15User-agent: Anomura
16User-agent: anthropic-ai
17User-agent: ApifyBot
18User-agent: ApifyWebsiteContentCrawler
19User-agent: Applebot
20User-agent: Applebot-Extended
21User-agent: Aranet-SearchBot
22User-agent: atlassian-bot
23User-agent: Awario
24User-agent: AzureAI-SearchBot
25User-agent: bedrockbot
26User-agent: bigsur.ai
27User-agent: Bravebot
28User-agent: Brightbot
29User-agent: Brightbot 1.0
30User-agent: BuddyBot
31User-agent: Bytespider
32User-agent: Channel3Bot
33User-agent: ChatGLM-Spider
34User-agent: ChatGPT Agent
35User-agent: ChatGPT-User
36User-agent: Claude-Code
37User-agent: Claude-SearchBot
38User-agent: Claude-User
39User-agent: Claude-Web
40User-agent: ClaudeBot
41User-agent: Cloudflare-AutoRAG
42User-agent: CloudVertexBot
43User-agent: Code
44User-agent: cohere-ai
45User-agent: cohere-training-data-crawler
46User-agent: Cotoyogi
47User-agent: CragCrawler
48User-agent: Crawl4AI
49User-agent: Crawlspace
50User-agent: Cursor
51User-agent: Datenbank Crawler
52User-agent: DeepSeekBot
53User-agent: Devin
54User-agent: Diffbot
55User-agent: DuckAssistBot
56User-agent: Echobot Bot
57User-agent: EchoboxBot
58User-agent: ExaBot
59User-agent: FacebookBot
60User-agent: facebookexternalhit
61User-agent: Factset_spyderbot
62User-agent: FirecrawlAgent
63User-agent: FriendlyCrawler
64User-agent: GeistHaus-PageFetcher
65User-agent: Gemini-Deep-Research
66User-agent: Google-Agent
67User-agent: Google-CloudVertexBot
68User-agent: Google-Extended
69User-agent: Google-Firebase
70User-agent: Google-Gemini-CLI
71User-agent: Google-NotebookLM
72User-agent: GoogleAgent-Mariner
73User-agent: GoogleAgent-URLContext
74User-agent: GoogleOther
75User-agent: GoogleOther-Image
76User-agent: GoogleOther-Video
77User-agent: GPTBot
78User-agent: HenkBot
79User-agent: iAskBot
80User-agent: iaskspider
81User-agent: iaskspider/2.0
82User-agent: IbouBot
83User-agent: ICC-Crawler
84User-agent: ImagesiftBot
85User-agent: imageSpider
86User-agent: img2dataset
87User-agent: ISSCyberRiskCrawler
88User-agent: kagi-fetcher
89User-agent: Kangaroo Bot
90User-agent: Kimi-User
91User-agent: KlaviyoAIBot
92User-agent: KunatoCrawler
93User-agent: laion-huggingface-processor
94User-agent: LAIONDownloader
95User-agent: LCC
96User-agent: LinerBot
97User-agent: Linguee Bot
98User-agent: LinkupBot
99User-agent: Manus-User
100User-agent: meta-externalagent
101User-agent: Meta-ExternalAgent
102User-agent: meta-externalfetcher
103User-agent: Meta-ExternalFetcher
104User-agent: meta-webindexer
105User-agent: MistralAI-User
106User-agent: MistralAI-User/1.0
107User-agent: Mozilla-Tabstack
108User-agent: MyCentralAIScraperBot
109User-agent: NagetBot
110User-agent: netEstate Imprint Crawler
111User-agent: newsai
112User-agent: NotebookLM
113User-agent: NovaAct
114User-agent: OAI-SearchBot
115User-agent: omgili
116User-agent: omgilibot
117User-agent: OpenAI
118User-agent: opencode
119User-agent: Operator
120User-agent: PanguBot
121User-agent: Panscient
122User-agent: panscient.com
123User-agent: Perplexity-User
124User-agent: PerplexityBot
125User-agent: PetalBot
126User-agent: PhindBot
127User-agent: Poggio-Citations
128User-agent: Poseidon Research Crawler
129User-agent: QualifiedBot
130User-agent: Querit-SearchBot
131User-agent: QueritBot
132User-agent: QuillBot
133User-agent: quillbot.com
134User-agent: SBIntuitionsBot
135User-agent: Scrapy
136User-agent: SemrushBot-OCOB
137User-agent: SemrushBot-SWA
138User-agent: Shap-User
139User-agent: ShapBot
140User-agent: Sidetrade indexer bot
141User-agent: Spider
142User-agent: TavilyBot
143User-agent: Terra Cotta
144User-agent: TerraCotta
145User-agent: Thinkbot
146User-agent: TikTokSpider
147User-agent: Timpibot
148User-agent: TongyiBot
149User-agent: Trae
150User-agent: TwinAgent
151User-agent: UseAI
152User-agent: VelenPublicWebCrawler
153User-agent: WARDBot
154User-agent: Webzio-Extended
155User-agent: webzio-extended
156User-agent: wpbot
157User-agent: WRTNBot
158User-agent: YaK
159User-agent: YandexAdditional
160User-agent: YandexAdditionalBot
161User-agent: YiyanBot
162User-agent: YouBot
163User-agent: ZanistaBot
164Disallow: /
165
166User-agent: *
167Disallow: /