DBLP / robots.txt
← back to dblp.org · change detected 12d ago (2026-09-07T11:45:01Z) · modified +95 −66
What changed
- Blocked user-agent ‘AddSearchBot’ entirely
- 🤖 AIBlocked AI2Bot (Allen Institute for AI) entirely
- Blocked user-agent ‘Ai2Bot-Dolma’ entirely
- Blocked user-agent ‘aiHitBot’ entirely
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘Andibot’ entirely
- 🤖 AIBlocked anthropic-ai (Anthropic) entirely
- Blocked user-agent ‘Applebot’ entirely
- 🤖 AIBlocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘Awario’ entirely
- Blocked user-agent ‘bedrockbot’ entirely
- Blocked user-agent ‘bigsur.ai’ entirely
- Blocked user-agent ‘Brightbot 1.0’ entirely
- 🤖 AIBlocked Bytespider (ByteDance) entirely
- 🤖 AIBlocked CCBot (Common Crawl) entirely
- Blocked user-agent ‘ChatGPT Agent’ entirely
- 🤖 AIBlocked ChatGPT-User (OpenAI) entirely
- 🤖 AIBlocked Claude-SearchBot (Anthropic) entirely
- 🤖 AIBlocked Claude-User (Anthropic) entirely
- 🤖 AIBlocked Claude-Web (Anthropic) entirely
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- Blocked user-agent ‘CloudVertexBot’ entirely
- 🤖 AIBlocked cohere-ai (Cohere) entirely
- Blocked user-agent ‘cohere-training-data-crawler’ entirely
- Blocked user-agent ‘Cotoyogi’ entirely
- Blocked user-agent ‘Crawlspace’ entirely
- Blocked user-agent ‘Datenbank Crawler’ entirely
- Blocked user-agent ‘Devin’ entirely
- 🤖 AIBlocked Diffbot (Diffbot) entirely
- 🤖 AIBlocked DuckAssistBot (DuckDuckGo) entirely
- Blocked user-agent ‘Echobot Bot’ entirely
- Blocked user-agent ‘EchoboxBot’ entirely
- 🤖 AIBlocked FacebookBot (Meta) entirely
- Blocked user-agent ‘facebookexternalhit’ entirely
- Blocked user-agent ‘Factset_spyderbot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘FriendlyCrawler’ entirely
- Blocked user-agent ‘Gemini-Deep-Research’ entirely
- Blocked user-agent ‘Google-CloudVertexBot’ entirely
- 🤖 AIBlocked Google-Extended (Google) entirely
- Blocked user-agent ‘GoogleAgent-Mariner’ entirely
- Blocked user-agent ‘GoogleOther’ entirely
- Blocked user-agent ‘GoogleOther-Image’ entirely
- Blocked user-agent ‘GoogleOther-Video’ entirely
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- Blocked user-agent ‘iaskspider/2.0’ entirely
- Blocked user-agent ‘ICC-Crawler’ entirely
- Blocked user-agent ‘ImagesiftBot’ entirely
- Blocked user-agent ‘img2dataset’ entirely
- Blocked user-agent ‘ISSCyberRiskCrawler’ entirely
- ... and 94 more change(s)
Diff old (2026-06-26T14:15:22Z) → new (2026-09-07T11:45:01Z)
| @@ -1,68 +1,97 @@ | ||
| 1 | - Sitemap: https://dblp.org/sitemap.xml | |
| 2 | - | |
| 1 | + User-agent: AddSearchBot | |
| 2 | + User-agent: AI2Bot | |
| 3 | + User-agent: Ai2Bot-Dolma | |
| 4 | + User-agent: aiHitBot | |
| 5 | + User-agent: Amazonbot | |
| 6 | + User-agent: Andibot | |
| 7 | + User-agent: anthropic-ai | |
| 8 | + User-agent: Applebot | |
| 9 | + User-agent: Applebot-Extended | |
| 10 | + User-agent: Awario | |
| 11 | + User-agent: bedrockbot | |
| 12 | + User-agent: bigsur.ai | |
| 13 | + User-agent: Brightbot 1.0 | |
| 14 | + User-agent: Bytespider | |
| 15 | + User-agent: CCBot | |
| 16 | + User-agent: ChatGPT Agent | |
| 17 | + User-agent: ChatGPT-User | |
| 18 | + User-agent: Claude-SearchBot | |
| 19 | + User-agent: Claude-User | |
| 20 | + User-agent: Claude-Web | |
| 21 | + User-agent: ClaudeBot | |
| 22 | + User-agent: CloudVertexBot | |
| 23 | + User-agent: cohere-ai | |
| 24 | + User-agent: cohere-training-data-crawler | |
| 25 | + User-agent: Cotoyogi | |
| 26 | + User-agent: Crawlspace | |
| 27 | + User-agent: Datenbank Crawler | |
| 28 | + User-agent: Devin | |
| 29 | + User-agent: Diffbot | |
| 30 | + User-agent: DuckAssistBot | |
| 31 | + User-agent: Echobot Bot | |
| 32 | + User-agent: EchoboxBot | |
| 33 | + User-agent: FacebookBot | |
| 34 | + User-agent: facebookexternalhit | |
| 35 | + User-agent: Factset_spyderbot | |
| 36 | + User-agent: FirecrawlAgent | |
| 37 | + User-agent: FriendlyCrawler | |
| 38 | + User-agent: Gemini-Deep-Research | |
| 39 | + User-agent: Google-CloudVertexBot | |
| 40 | + User-agent: Google-Extended | |
| 41 | + User-agent: GoogleAgent-Mariner | |
| 42 | + User-agent: GoogleOther | |
| 43 | + User-agent: GoogleOther-Image | |
| 44 | + User-agent: GoogleOther-Video | |
| 45 | + User-agent: GPTBot | |
| 46 | + User-agent: iaskspider/2.0 | |
| 47 | + User-agent: ICC-Crawler | |
| 48 | + User-agent: ImagesiftBot | |
| 49 | + User-agent: img2dataset | |
| 50 | + User-agent: ISSCyberRiskCrawler | |
| 51 | + User-agent: Kangaroo Bot | |
| 52 | + User-agent: LinerBot | |
| 53 | + User-agent: meta-externalagent | |
| 54 | + User-agent: Meta-ExternalAgent | |
| 55 | + User-agent: meta-externalfetcher | |
| 56 | + User-agent: Meta-ExternalFetcher | |
| 57 | + User-agent: MistralAI-User | |
| 58 | + User-agent: MistralAI-User/1.0 | |
| 59 | + User-agent: MyCentralAIScraperBot | |
| 60 | + User-agent: netEstate Imprint Crawler | |
| 61 | + User-agent: NovaAct | |
| 62 | + User-agent: OAI-SearchBot | |
| 63 | + User-agent: omgili | |
| 64 | + User-agent: omgilibot | |
| 65 | + User-agent: OpenAI | |
| 66 | + User-agent: Operator | |
| 67 | + User-agent: PanguBot | |
| 68 | + User-agent: Panscient | |
| 69 | + User-agent: panscient.com | |
| 70 | + User-agent: Perplexity-User | |
| 71 | + User-agent: PerplexityBot | |
| 72 | + User-agent: PetalBot | |
| 73 | + User-agent: PhindBot | |
| 74 | + User-agent: Poseidon Research Crawler | |
| 75 | + User-agent: QualifiedBot | |
| 76 | + User-agent: QuillBot | |
| 77 | + User-agent: quillbot.com | |
| 78 | + User-agent: SBIntuitionsBot | |
| 79 | + User-agent: Scrapy | |
| 80 | + User-agent: SemrushBot-OCOB | |
| 81 | + User-agent: SemrushBot-SWA | |
| 82 | + User-agent: Sidetrade indexer bot | |
| 83 | + User-agent: Thinkbot | |
| 84 | + User-agent: TikTokSpider | |
| 85 | + User-agent: Timpibot | |
| 86 | + User-agent: VelenPublicWebCrawler | |
| 87 | + User-agent: WARDBot | |
| 88 | + User-agent: Webzio-Extended | |
| 89 | + User-agent: wpbot | |
| 90 | + User-agent: YaK | |
| 91 | + User-agent: YandexAdditional | |
| 92 | + User-agent: YandexAdditionalBot | |
| 93 | + User-agent: YouBot | |
| 94 | + Disallow: / | |
| 3 | 95 | |
| 4 | 96 | User-agent: * |
| 5 | - | |
| 6 | - Crawl-delay: 4 | |
| 7 | - | |
| 8 | - Disallow: // | |
| 9 | - | |
| 10 | - Disallow: /cgi-bin | |
| 11 | - Disallow: /maps | |
| 12 | - | |
| 13 | - Allow: /db | |
| 14 | - Disallow: /db/indices | |
| 15 | - | |
| 16 | - Allow: /pers | |
| 17 | - Disallow: /pers/hb | |
| 18 | - Disallow: /pers/hg | |
| 19 | - Disallow: /pers/hj | |
| 20 | - Disallow: /pers/hk | |
| 21 | - Disallow: /pers/hx | |
| 22 | - Disallow: /pers/tb | |
| 23 | - Disallow: /pers/te | |
| 24 | - Disallow: /pers/tr | |
| 25 | - Disallow: /pers/xr | |
| 26 | - Disallow: /pers/xx | |
| 27 | - Disallow: /pers/xs | |
| 28 | - Disallow: /pers/xc | |
| 29 | - Disallow: /pers/xk | |
| 30 | - | |
| 31 | - Allow: /pid | |
| 32 | - | |
| 33 | - Allow: /rec | |
| 34 | - Disallow: /rec/bib | |
| 35 | - Disallow: /rec/ris | |
| 36 | - Disallow: /rec/nt | |
| 37 | - Disallow: /rec/rdf | |
| 38 | - Disallow: /rec/xml | |
| 39 | - | |
| 40 | - Disallow: /search | |
| 41 | - Disallow: /search/publ | |
| 42 | - Disallow: /search/author | |
| 43 | - Disallow: /search/venue | |
| 44 | - Disallow: /search/inst | |
| 45 | - Disallow: /search/yt | |
| 46 | - Allow: /search/$ | |
| 47 | - | |
| 48 | - Disallow: /lookup | |
| 49 | - Allow: /lookup/$ | |
| 50 | - | |
| 51 | - Disallow: /doi | |
| 52 | - Disallow: /isbn | |
| 53 | - Disallow: /issn | |
| 54 | - Disallow: /orcid | |
| 55 | - | |
| 56 | - Disallow: /*.bib | |
| 57 | - Disallow: /*.ris | |
| 58 | - Disallow: /*.nt | |
| 59 | - Disallow: /*.ttl | |
| 60 | - Disallow: /*.rdf | |
| 61 | - Disallow: /*.xml | |
| 62 | - Disallow: /*.json | |
| 63 | - | |
| 64 | - Disallow: /*view=bibtex | |
| 65 | - Disallow: /*view=keys | |
| 66 | - Disallow: /*view=group | |
| 67 | - Disallow: /*view=joint | |
| 68 | - | |
| 97 | + Disallow: / |