NeuralCrawl

DBLP / robots.txt snapshot

← back to dblp.org · fetched 2026-09-07T11:45:01Z (8d ago) · HTTP 200 · 2409 bytes · sha256 561ab459c867ddc9 · raw

final URL: https://dblp.org/robots.txt

1User-agent: AddSearchBot
2User-agent: AI2Bot
3User-agent: Ai2Bot-Dolma
4User-agent: aiHitBot
5User-agent: Amazonbot
6User-agent: Andibot
7User-agent: anthropic-ai
8User-agent: Applebot
9User-agent: Applebot-Extended
10User-agent: Awario
11User-agent: bedrockbot
12User-agent: bigsur.ai
13User-agent: Brightbot 1.0
14User-agent: Bytespider
15User-agent: CCBot
16User-agent: ChatGPT Agent
17User-agent: ChatGPT-User
18User-agent: Claude-SearchBot
19User-agent: Claude-User
20User-agent: Claude-Web
21User-agent: ClaudeBot
22User-agent: CloudVertexBot
23User-agent: cohere-ai
24User-agent: cohere-training-data-crawler
25User-agent: Cotoyogi
26User-agent: Crawlspace
27User-agent: Datenbank Crawler
28User-agent: Devin
29User-agent: Diffbot
30User-agent: DuckAssistBot
31User-agent: Echobot Bot
32User-agent: EchoboxBot
33User-agent: FacebookBot
34User-agent: facebookexternalhit
35User-agent: Factset_spyderbot
36User-agent: FirecrawlAgent
37User-agent: FriendlyCrawler
38User-agent: Gemini-Deep-Research
39User-agent: Google-CloudVertexBot
40User-agent: Google-Extended
41User-agent: GoogleAgent-Mariner
42User-agent: GoogleOther
43User-agent: GoogleOther-Image
44User-agent: GoogleOther-Video
45User-agent: GPTBot
46User-agent: iaskspider/2.0
47User-agent: ICC-Crawler
48User-agent: ImagesiftBot
49User-agent: img2dataset
50User-agent: ISSCyberRiskCrawler
51User-agent: Kangaroo Bot
52User-agent: LinerBot
53User-agent: meta-externalagent
54User-agent: Meta-ExternalAgent
55User-agent: meta-externalfetcher
56User-agent: Meta-ExternalFetcher
57User-agent: MistralAI-User
58User-agent: MistralAI-User/1.0
59User-agent: MyCentralAIScraperBot
60User-agent: netEstate Imprint Crawler
61User-agent: NovaAct
62User-agent: OAI-SearchBot
63User-agent: omgili
64User-agent: omgilibot
65User-agent: OpenAI
66User-agent: Operator
67User-agent: PanguBot
68User-agent: Panscient
69User-agent: panscient.com
70User-agent: Perplexity-User
71User-agent: PerplexityBot
72User-agent: PetalBot
73User-agent: PhindBot
74User-agent: Poseidon Research Crawler
75User-agent: QualifiedBot
76User-agent: QuillBot
77User-agent: quillbot.com
78User-agent: SBIntuitionsBot
79User-agent: Scrapy
80User-agent: SemrushBot-OCOB
81User-agent: SemrushBot-SWA
82User-agent: Sidetrade indexer bot
83User-agent: Thinkbot
84User-agent: TikTokSpider
85User-agent: Timpibot
86User-agent: VelenPublicWebCrawler
87User-agent: WARDBot
88User-agent: Webzio-Extended
89User-agent: wpbot
90User-agent: YaK
91User-agent: YandexAdditional
92User-agent: YandexAdditionalBot
93User-agent: YouBot
94Disallow: /
95
96User-agent: *
97Disallow: /