🇩🇪 DBLP

dblp.org · Academic & open research · rank #20 · CS bibliography · live robots.txt ↗

AI crawler access (latest snapshot, 3h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot

✅ChatGPT-User

✅OAI-SearchBot

✅ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

✅Google-Extended

✅Applebot-Extended

✅PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 1018 bytes · sha256 adec991215de · raw

Sitemap: https://dblp.org/sitemap.xml


User-agent: *

Crawl-delay: 4

Disallow: //

Disallow: /cgi-bin
Disallow: /maps

Allow:	  /db
Disallow: /db/indices

Allow:    /pers
Disallow: /pers/hb
Disallow: /pers/hg
Disallow: /pers/hj
Disallow: /pers/hk
Disallow: /pers/hx
Disallow: /pers/tb
Disallow: /pers/te
Disallow: /pers/tr
Disallow: /pers/xr
Disallow: /pers/xx
Disallow: /pers/xs
Disallow: /pers/xc
Disallow: /pers/xk

Allow:	  /pid

Allow:    /rec
Disallow: /rec/bib
Disallow: /rec/ris
Disallow: /rec/nt
Disallow: /rec/rdf
Disallow: /rec/xml

Disallow: /search
Disallow: /search/publ
Disallow: /search/author
Disallow: /search/venue
Disallow: /search/inst
Disallow: /search/yt
Allow:    /search/$

Disallow: /lookup
Allow:    /lookup/$

Disallow: /doi
Disallow: /isbn
Disallow: /issn
Disallow: /orcid

Disallow: /*.bib
Disallow: /*.ris
Disallow: /*.nt
Disallow: /*.ttl
Disallow: /*.rdf
Disallow: /*.xml
Disallow: /*.json

Disallow: /*view=bibtex
Disallow: /*view=keys
Disallow: /*view=group
Disallow: /*view=joint

Change history

initial 3h ago snapshot
- First snapshot of robots.txt archived