DBLP / robots.txt
← back to dblp.org · change detected 25d ago (2026-10-12T11:21:56Z) · modified +66 −165
What changed
- Removed all rules for user-agent ‘AddSearchBot’
- Removed all rules for user-agent ‘AgentTimes’
- 🤖 AIUnblocked AI2Bot (Allen Institute for AI) - rules removed
- Removed all rules for user-agent ‘AI2Bot-DeepResearchEval’
- Removed all rules for user-agent ‘Ai2Bot-Dolma’
- Removed all rules for user-agent ‘aiHitBot’
- Removed all rules for user-agent ‘AIWebIndex’
- Removed all rules for user-agent ‘amazon-kendra’
- Removed all rules for user-agent ‘amazon-QBusiness’
- 🤖 AIUnblocked Amazonbot (Amazon) - rules removed
- Removed all rules for user-agent ‘AmazonBuyForMe’
- Removed all rules for user-agent ‘Amzn-SearchBot’
- Removed all rules for user-agent ‘Amzn-User’
- Removed all rules for user-agent ‘Andibot’
- Removed all rules for user-agent ‘Anomura’
- 🤖 AIUnblocked anthropic-ai (Anthropic) - rules removed
- Removed all rules for user-agent ‘ApifyBot’
- Removed all rules for user-agent ‘ApifyWebsiteContentCrawler’
- Removed all rules for user-agent ‘Applebot’
- 🤖 AIUnblocked Applebot-Extended (Apple) - rules removed
- Removed all rules for user-agent ‘Aranet-SearchBot’
- Removed all rules for user-agent ‘atlassian-bot’
- Removed all rules for user-agent ‘Awario’
- Removed all rules for user-agent ‘AzureAI-SearchBot’
- Removed all rules for user-agent ‘bedrockbot’
- Removed all rules for user-agent ‘bigsur.ai’
- Removed all rules for user-agent ‘Bravebot’
- Removed all rules for user-agent ‘Brightbot’
- Removed all rules for user-agent ‘Brightbot 1.0’
- Removed all rules for user-agent ‘BuddyBot’
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- Removed all rules for user-agent ‘Channel3Bot’
- Removed all rules for user-agent ‘ChatGLM-Spider’
- Removed all rules for user-agent ‘ChatGPT Agent’
- 🤖 AIUnblocked ChatGPT-User (OpenAI) - rules removed
- Removed all rules for user-agent ‘Claude-Code’
- 🤖 AIUnblocked Claude-SearchBot (Anthropic) - rules removed
- 🤖 AIUnblocked Claude-User (Anthropic) - rules removed
- 🤖 AIUnblocked Claude-Web (Anthropic) - rules removed
- 🤖 AIUnblocked ClaudeBot (Anthropic) - rules removed
- Removed all rules for user-agent ‘Cloudflare-AutoRAG’
- Removed all rules for user-agent ‘CloudVertexBot’
- Removed all rules for user-agent ‘Code’
- 🤖 AIUnblocked cohere-ai (Cohere) - rules removed
- Removed all rules for user-agent ‘cohere-training-data-crawler’
- Removed all rules for user-agent ‘Cotoyogi’
- Removed all rules for user-agent ‘CragCrawler’
- Removed all rules for user-agent ‘Crawl4AI’
- Removed all rules for user-agent ‘Crawlspace’
- Removed all rules for user-agent ‘Cursor’
- ... and 163 more change(s)
Diff old (2026-09-09T18:36:35Z) → new (2026-10-12T11:21:56Z)
| @@ -1,167 +1,68 @@ | ||
| 1 | - User-agent: AddSearchBot | |
| 2 | - User-agent: AgentTimes | |
| 3 | - User-agent: AI2Bot | |
| 4 | - User-agent: AI2Bot-DeepResearchEval | |
| 5 | - User-agent: Ai2Bot-Dolma | |
| 6 | - User-agent: aiHitBot | |
| 7 | - User-agent: AIWebIndex | |
| 8 | - User-agent: amazon-kendra | |
| 9 | - User-agent: amazon-QBusiness | |
| 10 | - User-agent: Amazonbot | |
| 11 | - User-agent: AmazonBuyForMe | |
| 12 | - User-agent: Amzn-SearchBot | |
| 13 | - User-agent: Amzn-User | |
| 14 | - User-agent: Andibot | |
| 15 | - User-agent: Anomura | |
| 16 | - User-agent: anthropic-ai | |
| 17 | - User-agent: ApifyBot | |
| 18 | - User-agent: ApifyWebsiteContentCrawler | |
| 19 | - User-agent: Applebot | |
| 20 | - User-agent: Applebot-Extended | |
| 21 | - User-agent: Aranet-SearchBot | |
| 22 | - User-agent: atlassian-bot | |
| 23 | - User-agent: Awario | |
| 24 | - User-agent: AzureAI-SearchBot | |
| 25 | - User-agent: bedrockbot | |
| 26 | - User-agent: bigsur.ai | |
| 27 | - User-agent: Bravebot | |
| 28 | - User-agent: Brightbot | |
| 29 | - User-agent: Brightbot 1.0 | |
| 30 | - User-agent: BuddyBot | |
| 31 | - User-agent: Bytespider | |
| 32 | - User-agent: Channel3Bot | |
| 33 | - User-agent: ChatGLM-Spider | |
| 34 | - User-agent: ChatGPT Agent | |
| 35 | - User-agent: ChatGPT-User | |
| 36 | - User-agent: Claude-Code | |
| 37 | - User-agent: Claude-SearchBot | |
| 38 | - User-agent: Claude-User | |
| 39 | - User-agent: Claude-Web | |
| 40 | - User-agent: ClaudeBot | |
| 41 | - User-agent: Cloudflare-AutoRAG | |
| 42 | - User-agent: CloudVertexBot | |
| 43 | - User-agent: Code | |
| 44 | - User-agent: cohere-ai | |
| 45 | - User-agent: cohere-training-data-crawler | |
| 46 | - User-agent: Cotoyogi | |
| 47 | - User-agent: CragCrawler | |
| 48 | - User-agent: Crawl4AI | |
| 49 | - User-agent: Crawlspace | |
| 50 | - User-agent: Cursor | |
| 51 | - User-agent: Datenbank Crawler | |
| 52 | - User-agent: DeepSeekBot | |
| 53 | - User-agent: Devin | |
| 54 | - User-agent: Diffbot | |
| 55 | - User-agent: DuckAssistBot | |
| 56 | - User-agent: Echobot Bot | |
| 57 | - User-agent: EchoboxBot | |
| 58 | - User-agent: ExaBot | |
| 59 | - User-agent: FacebookBot | |
| 60 | - User-agent: facebookexternalhit | |
| 61 | - User-agent: Factset_spyderbot | |
| 62 | - User-agent: FirecrawlAgent | |
| 63 | - User-agent: FriendlyCrawler | |
| 64 | - User-agent: GeistHaus-PageFetcher | |
| 65 | - User-agent: Gemini-Deep-Research | |
| 66 | - User-agent: Google-Agent | |
| 67 | - User-agent: Google-CloudVertexBot | |
| 68 | - User-agent: Google-Extended | |
| 69 | - User-agent: Google-Firebase | |
| 70 | - User-agent: Google-Gemini-CLI | |
| 71 | - User-agent: Google-NotebookLM | |
| 72 | - User-agent: GoogleAgent-Mariner | |
| 73 | - User-agent: GoogleAgent-URLContext | |
| 74 | - User-agent: GoogleOther | |
| 75 | - User-agent: GoogleOther-Image | |
| 76 | - User-agent: GoogleOther-Video | |
| 77 | - User-agent: GPTBot | |
| 78 | - User-agent: HenkBot | |
| 79 | - User-agent: iAskBot | |
| 80 | - User-agent: iaskspider | |
| 81 | - User-agent: iaskspider/2.0 | |
| 82 | - User-agent: IbouBot | |
| 83 | - User-agent: ICC-Crawler | |
| 84 | - User-agent: ImagesiftBot | |
| 85 | - User-agent: imageSpider | |
| 86 | - User-agent: img2dataset | |
| 87 | - User-agent: ISSCyberRiskCrawler | |
| 88 | - User-agent: kagi-fetcher | |
| 89 | - User-agent: Kangaroo Bot | |
| 90 | - User-agent: Kimi-User | |
| 91 | - User-agent: KlaviyoAIBot | |
| 92 | - User-agent: KunatoCrawler | |
| 93 | - User-agent: laion-huggingface-processor | |
| 94 | - User-agent: LAIONDownloader | |
| 95 | - User-agent: LCC | |
| 96 | - User-agent: LinerBot | |
| 97 | - User-agent: Linguee Bot | |
| 98 | - User-agent: LinkupBot | |
| 99 | - User-agent: Manus-User | |
| 100 | - User-agent: meta-externalagent | |
| 101 | - User-agent: Meta-ExternalAgent | |
| 102 | - User-agent: meta-externalfetcher | |
| 103 | - User-agent: Meta-ExternalFetcher | |
| 104 | - User-agent: meta-webindexer | |
| 105 | - User-agent: MistralAI-User | |
| 106 | - User-agent: MistralAI-User/1.0 | |
| 107 | - User-agent: Mozilla-Tabstack | |
| 108 | - User-agent: MyCentralAIScraperBot | |
| 109 | - User-agent: NagetBot | |
| 110 | - User-agent: netEstate Imprint Crawler | |
| 111 | - User-agent: newsai | |
| 112 | - User-agent: NotebookLM | |
| 113 | - User-agent: NovaAct | |
| 114 | - User-agent: OAI-SearchBot | |
| 115 | - User-agent: omgili | |
| 116 | - User-agent: omgilibot | |
| 117 | - User-agent: OpenAI | |
| 118 | - User-agent: opencode | |
| 119 | - User-agent: Operator | |
| 120 | - User-agent: PanguBot | |
| 121 | - User-agent: Panscient | |
| 122 | - User-agent: panscient.com | |
| 123 | - User-agent: Perplexity-User | |
| 124 | - User-agent: PerplexityBot | |
| 125 | - User-agent: PetalBot | |
| 126 | - User-agent: PhindBot | |
| 127 | - User-agent: Poggio-Citations | |
| 128 | - User-agent: Poseidon Research Crawler | |
| 129 | - User-agent: QualifiedBot | |
| 130 | - User-agent: Querit-SearchBot | |
| 131 | - User-agent: QueritBot | |
| 132 | - User-agent: QuillBot | |
| 133 | - User-agent: quillbot.com | |
| 134 | - User-agent: SBIntuitionsBot | |
| 135 | - User-agent: Scrapy | |
| 136 | - User-agent: SemrushBot-OCOB | |
| 137 | - User-agent: SemrushBot-SWA | |
| 138 | - User-agent: Shap-User | |
| 139 | - User-agent: ShapBot | |
| 140 | - User-agent: Sidetrade indexer bot | |
| 141 | - User-agent: Spider | |
| 142 | - User-agent: TavilyBot | |
| 143 | - User-agent: Terra Cotta | |
| 144 | - User-agent: TerraCotta | |
| 145 | - User-agent: Thinkbot | |
| 146 | - User-agent: TikTokSpider | |
| 147 | - User-agent: Timpibot | |
| 148 | - User-agent: TongyiBot | |
| 149 | - User-agent: Trae | |
| 150 | - User-agent: TwinAgent | |
| 151 | - User-agent: UseAI | |
| 152 | - User-agent: VelenPublicWebCrawler | |
| 153 | - User-agent: WARDBot | |
| 154 | - User-agent: Webzio-Extended | |
| 155 | - User-agent: webzio-extended | |
| 156 | - User-agent: wpbot | |
| 157 | - User-agent: WRTNBot | |
| 158 | - User-agent: YaK | |
| 159 | - User-agent: YandexAdditional | |
| 160 | - User-agent: YandexAdditionalBot | |
| 161 | - User-agent: YiyanBot | |
| 162 | - User-agent: YouBot | |
| 163 | - User-agent: ZanistaBot | |
| 164 | - Disallow: / | |
| 1 | + Sitemap: https://dblp.org/sitemap.xml | |
| 2 | + | |
| 165 | 3 | |
| 166 | 4 | User-agent: * |
| 167 | - Disallow: / | |
| 5 | + | |
| 6 | + Crawl-delay: 4 | |
| 7 | + | |
| 8 | + Disallow: // | |
| 9 | + | |
| 10 | + Disallow: /cgi-bin | |
| 11 | + Disallow: /maps | |
| 12 | + | |
| 13 | + Allow: /db | |
| 14 | + Disallow: /db/indices | |
| 15 | + | |
| 16 | + Allow: /pers | |
| 17 | + Disallow: /pers/hb | |
| 18 | + Disallow: /pers/hg | |
| 19 | + Disallow: /pers/hj | |
| 20 | + Disallow: /pers/hk | |
| 21 | + Disallow: /pers/hx | |
| 22 | + Disallow: /pers/tb | |
| 23 | + Disallow: /pers/te | |
| 24 | + Disallow: /pers/tr | |
| 25 | + Disallow: /pers/xr | |
| 26 | + Disallow: /pers/xx | |
| 27 | + Disallow: /pers/xs | |
| 28 | + Disallow: /pers/xc | |
| 29 | + Disallow: /pers/xk | |
| 30 | + | |
| 31 | + Allow: /pid | |
| 32 | + | |
| 33 | + Allow: /rec | |
| 34 | + Disallow: /rec/bib | |
| 35 | + Disallow: /rec/ris | |
| 36 | + Disallow: /rec/nt | |
| 37 | + Disallow: /rec/rdf | |
| 38 | + Disallow: /rec/xml | |
| 39 | + | |
| 40 | + Disallow: /search | |
| 41 | + Disallow: /search/publ | |
| 42 | + Disallow: /search/author | |
| 43 | + Disallow: /search/venue | |
| 44 | + Disallow: /search/inst | |
| 45 | + Disallow: /search/yt | |
| 46 | + Allow: /search/$ | |
| 47 | + | |
| 48 | + Disallow: /lookup | |
| 49 | + Allow: /lookup/$ | |
| 50 | + | |
| 51 | + Disallow: /doi | |
| 52 | + Disallow: /isbn | |
| 53 | + Disallow: /issn | |
| 54 | + Disallow: /orcid | |
| 55 | + | |
| 56 | + Disallow: /*.bib | |
| 57 | + Disallow: /*.ris | |
| 58 | + Disallow: /*.nt | |
| 59 | + Disallow: /*.ttl | |
| 60 | + Disallow: /*.rdf | |
| 61 | + Disallow: /*.xml | |
| 62 | + Disallow: /*.json | |
| 63 | + | |
| 64 | + Disallow: /*view=bibtex | |
| 65 | + Disallow: /*view=keys | |
| 66 | + Disallow: /*view=group | |
| 67 | + Disallow: /*view=joint | |
| 68 | + |