NeuralCrawl

Europe PMC / robots.txt

← back to europepmc.org · change detected 19d ago (2026-07-16T23:56:01Z) · modified +98 −7

What changed

Diff old (2026-07-16T17:46:03Z) → new (2026-07-16T23:56:01Z)

@@ -3,16 +3,15 @@
33  # Google
44  User-agent: Googlebot
55  Disallow: /classic
6- #Disallow: /*/articles/*/pdf/
7- #Disallow: /articles/*/pdf/
8- #Disallow: /articles/*?pdf=render
9- #Disallow: /articles/*?*blobtype=pdf*
10- Disallow: /advancedsearch
116  Disallow: /feedback/*
127  Disallow: /*?wicket:interface*
13- #Disallow: /search/*
148  Disallow: /*?europe_pmc_extredirect=*
9+ Disallow: /ftp/oa/*
10+ Disallow: /Funders/rss/*
11+ Disallow: /exception*
12+ Disallow: /*/*jsessionid*
1513 
14+ #
1615  # Yahoo
1716  User-agent: Slurp
1817  Disallow: /classic
@@ -24,9 +23,24 @@
2423  Disallow: /feedback/*
2524  Disallow: /*?wicket:interface*
2625  Disallow: /search/*
26+ Disallow: /search?*
27+ Disallow: /abstract/*
2728  Disallow: /*?europe_pmc_extredirect=*
29+ Disallow: /ftp/oa/*
30+ Disallow: /exception/*
31+ Disallow: /Funders/rss/*
2832 
2933  # Bing
34+ User-agent: bingbot
35+ Disallow: /classic
36+ Disallow: /feedback/*
37+ Disallow: /*?wicket:interface*
38+ Disallow: /*?europe_pmc_extredirect=*
39+ Disallow: /ftp/oa/*
40+ Disallow: /Funders/rss/*
41+ Disallow: /exception*
42+ Disallow: /*/*jsessionid*
43+
3044  User-agent: MSNbot
3145  Disallow: /classic
3246  Disallow: /*/articles/*/pdf/
@@ -36,8 +50,81 @@
3650  Disallow: /advancedsearch
3751  Disallow: /feedback/*
3852  Disallow: /*?wicket:interface*
53+ Disallow: /ftp/oa/*
3954  Disallow: /search/*
55+ Disallow: /search?*
56+ Disallow: /abstract/*
4057  Disallow: /*?europe_pmc_extredirect=*
58+ Disallow: /Funders/rss/*
59+ Disallow: /exception/*
60+
61+ User-agent: baiduspider
62+ Disallow: /classic
63+ Disallow: /*/articles/*/pdf/
64+ Disallow: /articles/*/pdf/
65+ Disallow: /articles/*?pdf=render
66+ Disallow: /articles/*?*blobtype=pdf*
67+ Disallow: /advancedsearch
68+ Disallow: /feedback/*
69+ Disallow: /*?wicket:interface*
70+ Disallow: /ftp/oa/*
71+ Disallow: /search/*
72+ Disallow: /search?*
73+ Disallow: /abstract/*
74+ Disallow: /*?europe_pmc_extredirect=*
75+ Disallow: /Funders/rss/*
76+ Disallow: /exception/*
77+
78+ User-agent: OAI-SearchBot
79+ Disallow: /classic
80+ Disallow: /feedback/*
81+ Disallow: /*?wicket:interface*
82+ Disallow: /*?europe_pmc_extredirect=*
83+ Disallow: /ftp/oa/*
84+ Disallow: /Funders/rss/*
85+ Disallow: /exception*
86+ Disallow: /*/*jsessionid*
87+
88+ User-agent: ChatGPT-User
89+ Disallow: /classic
90+ Disallow: /feedback/*
91+ Disallow: /*?wicket:interface*
92+ Disallow: /*?europe_pmc_extredirect=*
93+ Disallow: /ftp/oa/*
94+ Disallow: /Funders/rss/*
95+ Disallow: /exception*
96+ Disallow: /*/*jsessionid*
97+
98+ User-agent: GPTBot
99+ Disallow: /classic
100+ Disallow: /feedback/*
101+ Disallow: /*?wicket:interface*
102+ Disallow: /*?europe_pmc_extredirect=*
103+ Disallow: /ftp/oa/*
104+ Disallow: /Funders/rss/*
105+ Disallow: /exception*
106+ Disallow: /*/*jsessionid*
107+
108+ User-agent: PerplexityBot
109+ Disallow: /classic
110+ Disallow: /feedback/*
111+ Disallow: /*?wicket:interface*
112+ Disallow: /*?europe_pmc_extredirect=*
113+ Disallow: /ftp/oa/*
114+ Disallow: /Funders/rss/*
115+ Disallow: /exception*
116+ Disallow: /*/*jsessionid*
117+
118+ User-agent: Perplexity‑User
119+ Disallow: /classic
120+ Disallow: /feedback/*
121+ Disallow: /*?wicket:interface*
122+ Disallow: /*?europe_pmc_extredirect=*
123+ Disallow: /ftp/oa/*
124+ Disallow: /Funders/rss/*
125+ Disallow: /exception*
126+ Disallow: /*/*jsessionid*
127+
41128 
42129  # For all other bots
43130  User-agent: *
@@ -45,6 +132,10 @@
45132 
46133  Sitemap: http://europepmc.org/sitemap_abstract_index.xml
47134  Sitemap: http://europepmc.org/sitemap_fulltext_index.xml
48- Crawl-delay: 10
135+ Sitemap: http://europepmc.org/sitemap_authors_index.xml
136+ Sitemap: http://europepmc.org/sitemap_funders_index.xml
137+ Sitemap: http://europepmc.org/sitemap_search_index.xml
138+ Sitemap: http://europepmc.org/sitemap_grants_index.xml
139+ Crawl-delay: 5
49140 
50141