NeuralCrawl

Europe PMC / robots.txt

← back to europepmc.org · change detected 11d ago (2026-09-07T17:46:37Z) · modified +141 −0

What changed

Diff old (2026-08-29T14:58:06Z) → new (2026-09-07T17:46:37Z)

@@ -59,3 +59,144 @@
5959 
6060  # END Cloudflare Managed Content
6161 
62+ # UKPMC robots.txt
63+
64+ # Google
65+ User-agent: Googlebot
66+ Disallow: /classic
67+ Disallow: /feedback/*
68+ Disallow: /*?wicket:interface*
69+ Disallow: /*?europe_pmc_extredirect=*
70+ Disallow: /ftp/oa/*
71+ Disallow: /Funders/rss/*
72+ Disallow: /exception*
73+ Disallow: /*/*jsessionid*
74+
75+ #
76+ # Yahoo
77+ User-agent: Slurp
78+ Disallow: /classic
79+ Disallow: /*/articles/*/pdf/
80+ Disallow: /articles/*/pdf/
81+ Disallow: /articles/*?pdf=render
82+ Disallow: /articles/*?*blobtype=pdf*
83+ Disallow: /advancedsearch
84+ Disallow: /feedback/*
85+ Disallow: /*?wicket:interface*
86+ Disallow: /search/*
87+ Disallow: /search?*
88+ Disallow: /abstract/*
89+ Disallow: /*?europe_pmc_extredirect=*
90+ Disallow: /ftp/oa/*
91+ Disallow: /exception/*
92+ Disallow: /Funders/rss/*
93+
94+ # Bing
95+ User-agent: bingbot
96+ Disallow: /classic
97+ Disallow: /feedback/*
98+ Disallow: /*?wicket:interface*
99+ Disallow: /*?europe_pmc_extredirect=*
100+ Disallow: /ftp/oa/*
101+ Disallow: /Funders/rss/*
102+ Disallow: /exception*
103+ Disallow: /*/*jsessionid*
104+
105+ User-agent: MSNbot
106+ Disallow: /classic
107+ Disallow: /*/articles/*/pdf/
108+ Disallow: /articles/*/pdf/
109+ Disallow: /articles/*?pdf=render
110+ Disallow: /articles/*?*blobtype=pdf*
111+ Disallow: /advancedsearch
112+ Disallow: /feedback/*
113+ Disallow: /*?wicket:interface*
114+ Disallow: /ftp/oa/*
115+ Disallow: /search/*
116+ Disallow: /search?*
117+ Disallow: /abstract/*
118+ Disallow: /*?europe_pmc_extredirect=*
119+ Disallow: /Funders/rss/*
120+ Disallow: /exception/*
121+
122+ User-agent: baiduspider
123+ Disallow: /classic
124+ Disallow: /*/articles/*/pdf/
125+ Disallow: /articles/*/pdf/
126+ Disallow: /articles/*?pdf=render
127+ Disallow: /articles/*?*blobtype=pdf*
128+ Disallow: /advancedsearch
129+ Disallow: /feedback/*
130+ Disallow: /*?wicket:interface*
131+ Disallow: /ftp/oa/*
132+ Disallow: /search/*
133+ Disallow: /search?*
134+ Disallow: /abstract/*
135+ Disallow: /*?europe_pmc_extredirect=*
136+ Disallow: /Funders/rss/*
137+ Disallow: /exception/*
138+
139+ User-agent: OAI-SearchBot
140+ Disallow: /classic
141+ Disallow: /feedback/*
142+ Disallow: /*?wicket:interface*
143+ Disallow: /*?europe_pmc_extredirect=*
144+ Disallow: /ftp/oa/*
145+ Disallow: /Funders/rss/*
146+ Disallow: /exception*
147+ Disallow: /*/*jsessionid*
148+
149+ User-agent: ChatGPT-User
150+ Disallow: /classic
151+ Disallow: /feedback/*
152+ Disallow: /*?wicket:interface*
153+ Disallow: /*?europe_pmc_extredirect=*
154+ Disallow: /ftp/oa/*
155+ Disallow: /Funders/rss/*
156+ Disallow: /exception*
157+ Disallow: /*/*jsessionid*
158+
159+ User-agent: GPTBot
160+ Disallow: /classic
161+ Disallow: /feedback/*
162+ Disallow: /*?wicket:interface*
163+ Disallow: /*?europe_pmc_extredirect=*
164+ Disallow: /ftp/oa/*
165+ Disallow: /Funders/rss/*
166+ Disallow: /exception*
167+ Disallow: /*/*jsessionid*
168+
169+ User-agent: PerplexityBot
170+ Disallow: /classic
171+ Disallow: /feedback/*
172+ Disallow: /*?wicket:interface*
173+ Disallow: /*?europe_pmc_extredirect=*
174+ Disallow: /ftp/oa/*
175+ Disallow: /Funders/rss/*
176+ Disallow: /exception*
177+ Disallow: /*/*jsessionid*
178+
179+ User-agent: Perplexity‑User
180+ Disallow: /classic
181+ Disallow: /feedback/*
182+ Disallow: /*?wicket:interface*
183+ Disallow: /*?europe_pmc_extredirect=*
184+ Disallow: /ftp/oa/*
185+ Disallow: /Funders/rss/*
186+ Disallow: /exception*
187+ Disallow: /*/*jsessionid*
188+
189+
190+ # For all other bots
191+ User-agent: *
192+ Disallow: /
193+
194+ Sitemap: http://europepmc.org/sitemap_abstract_index.xml
195+ Sitemap: http://europepmc.org/sitemap_fulltext_index.xml
196+ Sitemap: http://europepmc.org/sitemap_authors_index.xml
197+ Sitemap: http://europepmc.org/sitemap_funders_index.xml
198+ Sitemap: http://europepmc.org/sitemap_search_index.xml
199+ Sitemap: http://europepmc.org/sitemap_grants_index.xml
200+ Crawl-delay: 5
201+
202+