Elsevier / robots.txt snapshot
← back to elsevier.com · fetched 2026-06-20T01:10:31Z (18h ago) · HTTP 200 · 490 bytes · sha256 4e72b2758f3b723f · raw
final URL: https://www.elsevier.com/robots.txt
| 1 | # robots.txt file for https://www.elsevier.com |
| 2 | |
| 3 | # Disallow |
| 4 | User-agent: FAST Enterprise Crawler 6 / Scirus |
| 5 | Disallow: / |
| 6 | |
| 7 | User-agent: innosense/Nutch-1.0 |
| 8 | Disallow: / |
| 9 | |
| 10 | User-agent: Sogou web spider/4.0 |
| 11 | Disallow: / |
| 12 | |
| 13 | User-agent: Xenu Link Sleuth/1.3.8 |
| 14 | Disallow: / |
| 15 | |
| 16 | User-agent: discoverybot/2.0 |
| 17 | Disallow: / |
| 18 | |
| 19 | User-agent: YoudaoBot/1.0 |
| 20 | Disallow: / |
| 21 | |
| 22 | User-agent: Sogou web spider/3.0 |
| 23 | Disallow: / |
| 24 | |
| 25 | |
| 26 | # Allow |
| 27 | User-agent: * |
| 28 | Allow: / |
| 29 | |
| 30 | # Sitemaps |
| 31 | Sitemap: https://www.elsevier.com/sitemaps/index.xml |