NeuralCrawl

πŸ‡¬πŸ‡§ Europe PMC

europepmc.org · Academic & open research · rank #15 · Research repository · live robots.txt ↗

AI crawler access (latest snapshot, 3h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 3282 bytes · sha256 bf63b304ea4b · raw

# UKPMC robots.txt

# Google
User-agent: Googlebot
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*

#
# Yahoo
User-agent: Slurp
Disallow: /classic
Disallow: /*/articles/*/pdf/
Disallow: /articles/*/pdf/
Disallow: /articles/*?pdf=render
Disallow: /articles/*?*blobtype=pdf*
Disallow: /advancedsearch
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /search/*
Disallow: /search?*
Disallow: /abstract/*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /exception/*
Disallow: /Funders/rss/*

# Bing
User-agent: bingbot
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*

User-agent: MSNbot
Disallow: /classic
Disallow: /*/articles/*/pdf/
Disallow: /articles/*/pdf/
Disallow: /articles/*?pdf=render
Disallow: /articles/*?*blobtype=pdf*
Disallow: /advancedsearch
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /ftp/oa/*
Disallow: /search/*
Disallow: /search?*
Disallow: /abstract/*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /Funders/rss/*
Disallow: /exception/*

User-agent: baiduspider
Disallow: /classic
Disallow: /*/articles/*/pdf/
Disallow: /articles/*/pdf/
Disallow: /articles/*?pdf=render
Disallow: /articles/*?*blobtype=pdf*
Disallow: /advancedsearch
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /ftp/oa/*
Disallow: /search/*
Disallow: /search?*
Disallow: /abstract/*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /Funders/rss/*
Disallow: /exception/*

User-agent: OAI-SearchBot
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*

User-agent: ChatGPT-User
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*

User-agent: GPTBot
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*

User-agent: PerplexityBot
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*

User-agent: Perplexity‑User
Disallow: /classic
Disallow: /feedback/*
Disallow: /*?wicket:interface*
Disallow: /*?europe_pmc_extredirect=*
Disallow: /ftp/oa/*
Disallow: /Funders/rss/*
Disallow: /exception*
Disallow: /*/*jsessionid*


# For all other bots
User-agent: *
Disallow: /

Sitemap: http://europepmc.org/sitemap_abstract_index.xml
Sitemap: http://europepmc.org/sitemap_fulltext_index.xml
Sitemap: http://europepmc.org/sitemap_authors_index.xml
Sitemap: http://europepmc.org/sitemap_funders_index.xml
Sitemap: http://europepmc.org/sitemap_search_index.xml
Sitemap: http://europepmc.org/sitemap_grants_index.xml
Crawl-delay: 5

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived