NeuralCrawl

Europe PMC / robots.txt

← back to europepmc.org · change detected 24d ago (2026-08-25T13:40:59Z) · modified +48 −128

What changed

Diff old (2026-07-16T23:56:01Z) → new (2026-08-25T13:40:59Z)

@@ -1,141 +1,61 @@
1- # UKPMC robots.txt
1+ # As a condition of accessing this website, you agree to abide by the following
2+ # content signals:
23 
3- # Google
4- User-agent: Googlebot
5- Disallow: /classic
6- Disallow: /feedback/*
7- Disallow: /*?wicket:interface*
8- Disallow: /*?europe_pmc_extredirect=*
9- Disallow: /ftp/oa/*
10- Disallow: /Funders/rss/*
11- Disallow: /exception*
12- Disallow: /*/*jsessionid*
4+ # (a) If a Content-Signal = yes, you may collect content for the corresponding
5+ # use.
6+ # (b) If a Content-Signal = no, you may not collect content for the
7+ # corresponding use.
8+ # (c) If the website operator does not include a Content-Signal for a
9+ # corresponding use, the website operator neither grants nor restricts
10+ # permission via Content-Signal with respect to the corresponding use.
1311 
14- #
15- # Yahoo
16- User-agent: Slurp
17- Disallow: /classic
18- Disallow: /*/articles/*/pdf/
19- Disallow: /articles/*/pdf/
20- Disallow: /articles/*?pdf=render
21- Disallow: /articles/*?*blobtype=pdf*
22- Disallow: /advancedsearch
23- Disallow: /feedback/*
24- Disallow: /*?wicket:interface*
25- Disallow: /search/*
26- Disallow: /search?*
27- Disallow: /abstract/*
28- Disallow: /*?europe_pmc_extredirect=*
29- Disallow: /ftp/oa/*
30- Disallow: /exception/*
31- Disallow: /Funders/rss/*
12+ # The content signals and their meanings are:
3213 
33- # Bing
34- User-agent: bingbot
35- Disallow: /classic
36- Disallow: /feedback/*
37- Disallow: /*?wicket:interface*
38- Disallow: /*?europe_pmc_extredirect=*
39- Disallow: /ftp/oa/*
40- Disallow: /Funders/rss/*
41- Disallow: /exception*
42- Disallow: /*/*jsessionid*
14+ # search: building a search index and providing search results (e.g., returning
15+ # hyperlinks and short excerpts from your website's contents). Search does not
16+ # include providing AI-generated search summaries.
17+ # ai-input: inputting content into one or more AI models (e.g., retrieval
18+ # augmented generation, grounding, or other real-time taking of content for
19+ # generative AI search answers).
20+ # ai-train: training or fine-tuning AI models.
21+ # use: how AI systems may consume the content (immediate, reference, or full).
4322 
44- User-agent: MSNbot
45- Disallow: /classic
46- Disallow: /*/articles/*/pdf/
47- Disallow: /articles/*/pdf/
48- Disallow: /articles/*?pdf=render
49- Disallow: /articles/*?*blobtype=pdf*
50- Disallow: /advancedsearch
51- Disallow: /feedback/*
52- Disallow: /*?wicket:interface*
53- Disallow: /ftp/oa/*
54- Disallow: /search/*
55- Disallow: /search?*
56- Disallow: /abstract/*
57- Disallow: /*?europe_pmc_extredirect=*
58- Disallow: /Funders/rss/*
59- Disallow: /exception/*
23+ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24+ # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25+ # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
6026 
61- User-agent: baiduspider
62- Disallow: /classic
63- Disallow: /*/articles/*/pdf/
64- Disallow: /articles/*/pdf/
65- Disallow: /articles/*?pdf=render
66- Disallow: /articles/*?*blobtype=pdf*
67- Disallow: /advancedsearch
68- Disallow: /feedback/*
69- Disallow: /*?wicket:interface*
70- Disallow: /ftp/oa/*
71- Disallow: /search/*
72- Disallow: /search?*
73- Disallow: /abstract/*
74- Disallow: /*?europe_pmc_extredirect=*
75- Disallow: /Funders/rss/*
76- Disallow: /exception/*
27+ # BEGIN Cloudflare Managed content
7728 
78- User-agent: OAI-SearchBot
79- Disallow: /classic
80- Disallow: /feedback/*
81- Disallow: /*?wicket:interface*
82- Disallow: /*?europe_pmc_extredirect=*
83- Disallow: /ftp/oa/*
84- Disallow: /Funders/rss/*
85- Disallow: /exception*
86- Disallow: /*/*jsessionid*
29+ User-agent: *
30+ Content-Signal: search=yes,ai-train=no,use=reference
31+ Allow: /
8732 
88- User-agent: ChatGPT-User
89- Disallow: /classic
90- Disallow: /feedback/*
91- Disallow: /*?wicket:interface*
92- Disallow: /*?europe_pmc_extredirect=*
93- Disallow: /ftp/oa/*
94- Disallow: /Funders/rss/*
95- Disallow: /exception*
96- Disallow: /*/*jsessionid*
33+ User-agent: Amazonbot
34+ Disallow: /
35+
36+ User-agent: Applebot-Extended
37+ Disallow: /
38+
39+ User-agent: Bytespider
40+ Disallow: /
41+
42+ User-agent: CCBot
43+ Disallow: /
44+
45+ User-agent: ClaudeBot
46+ Disallow: /
47+
48+ User-agent: CloudflareBrowserRenderingCrawler
49+ Disallow: /
50+
51+ User-agent: Google-Extended
52+ Disallow: /
9753 
9854  User-agent: GPTBot
99- Disallow: /classic
100- Disallow: /feedback/*
101- Disallow: /*?wicket:interface*
102- Disallow: /*?europe_pmc_extredirect=*
103- Disallow: /ftp/oa/*
104- Disallow: /Funders/rss/*
105- Disallow: /exception*
106- Disallow: /*/*jsessionid*
107-
108- User-agent: PerplexityBot
109- Disallow: /classic
110- Disallow: /feedback/*
111- Disallow: /*?wicket:interface*
112- Disallow: /*?europe_pmc_extredirect=*
113- Disallow: /ftp/oa/*
114- Disallow: /Funders/rss/*
115- Disallow: /exception*
116- Disallow: /*/*jsessionid*
117-
118- User-agent: Perplexity‑User
119- Disallow: /classic
120- Disallow: /feedback/*
121- Disallow: /*?wicket:interface*
122- Disallow: /*?europe_pmc_extredirect=*
123- Disallow: /ftp/oa/*
124- Disallow: /Funders/rss/*
125- Disallow: /exception*
126- Disallow: /*/*jsessionid*
127-
128-
129- # For all other bots
130- User-agent: *
13155  Disallow: /
13256 
133- Sitemap: http://europepmc.org/sitemap_abstract_index.xml
134- Sitemap: http://europepmc.org/sitemap_fulltext_index.xml
135- Sitemap: http://europepmc.org/sitemap_authors_index.xml
136- Sitemap: http://europepmc.org/sitemap_funders_index.xml
137- Sitemap: http://europepmc.org/sitemap_search_index.xml
138- Sitemap: http://europepmc.org/sitemap_grants_index.xml
139- Crawl-delay: 5
57+ User-agent: meta-externalagent
58+ Disallow: /
14059 
60+ # END Cloudflare Managed Content
14161