NeuralCrawl

Fresenius / robots.txt

← back to fresenius.com · change detected 29d ago (2026-10-08T22:07:03Z) · modified +6 −1

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-10-08T22:07:03Z)

@@ -19,20 +19,24 @@
1919  Allow: /core/*.css?
2020  Allow: /core/*.js$
2121  Allow: /core/*.js?
22+ Allow: /core/*.avif
2223  Allow: /core/*.gif
2324  Allow: /core/*.jpg
2425  Allow: /core/*.jpeg
2526  Allow: /core/*.png
2627  Allow: /core/*.svg
28+ Allow: /core/*.webp
2729  Allow: /profiles/*.css$
2830  Allow: /profiles/*.css?
2931  Allow: /profiles/*.js$
3032  Allow: /profiles/*.js?
33+ Allow: /profiles/*.avif
3134  Allow: /profiles/*.gif
3235  Allow: /profiles/*.jpg
3336  Allow: /profiles/*.jpeg
3437  Allow: /profiles/*.png
3538  Allow: /profiles/*.svg
39+ Allow: /profiles/*.webp
3640  # Directories
3741  Disallow: /core/
3842  Disallow: /profiles/
@@ -46,13 +50,13 @@
4650  Disallow: /modules/README.txt
4751  Disallow: /sites/README.txt
4852  Disallow: /themes/README.txt
49- Disallow: /web.config
5053  # Paths (clean URLs)
5154  Disallow: /admin/
5255  Disallow: /comment/reply/
5356  Disallow: /filter/tips
5457  Disallow: /node/add/
5558  Disallow: /search/
59+ Disallow: /search?
5660  Disallow: /user/register
5761  Disallow: /user/password
5862  Disallow: /user/login
@@ -65,6 +69,7 @@
6569  Disallow: /index.php/filter/tips
6670  Disallow: /index.php/node/add/
6771  Disallow: /index.php/search/
72+ Disallow: /index.php/search?
6873  Disallow: /index.php/user/password
6974  Disallow: /index.php/user/register
7075  Disallow: /index.php/user/login