NeuralCrawl

Regeneron Pharmaceuticals / robots.txt snapshot

← back to regeneron.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2249 bytes · sha256 b950e714504e0376 · raw

final URL: https://www.regeneron.com/robots.txt

1#updated 05/14/2026
2
3# AI / LLM Crawlers
4User-agent: GPTBot
5User-agent: ChatGPT-User
6User-agent: Google-Extended
7User-agent: ClaudeBot
8User-agent: PerplexityBot
9User-agent: Perplexity-User
10User-agent: Bytespider
11User-agent: Meta-ExternalAgent
12User-agent: Meta-ExternalFetcher
13User-agent: Amazonbot
14User-agent: Applebot-Extended
15User-agent: CCBot
16User-agent: Diffbot
17
18Disallow: /about/leadership$
19Disallow: /about/leadership/$
20Disallow: /node/*
21Disallow: /search-results*
22# Files
23Disallow: /CHANGELOG.txt
24Disallow: /cron.php
25Disallow: /INSTALL.mysql.txt
26Disallow: /INSTALL.pgsql.txt
27Disallow: /INSTALL.sqlite.txt
28Disallow: /install.php
29Disallow: /INSTALL.txt
30Disallow: /LICENSE.txt
31Disallow: /MAINTAINERS.txt
32Disallow: /update.php
33Disallow: /UPGRADE.txt
34Disallow: /xmlrpc.php
35# Paths (clean URLs)
36Disallow: /admin
37Disallow: /comment/reply
38Disallow: /filter/tips
39Disallow: /node/add
40Disallow: /search
41Disallow: /user/register
42Disallow: /user/password
43Disallow: /user/login
44Disallow: /user/logout
45# Paths (no clean URLs)
46Disallow: /?q=admin
47Disallow: /?q=comment/reply
48Disallow: /?q=filter/tips
49Disallow: /?q=node/add
50Disallow: /?q=search
51Disallow: /?q=user/password
52Disallow: /?q=user/register
53Disallow: /?q=user/login
54Disallow: /?q=user/logout
55
56User-agent: *
57
58Disallow: /node/*
59Disallow: /search-results*
60# Files
61Disallow: /CHANGELOG.txt
62Disallow: /cron.php
63Disallow: /INSTALL.mysql.txt
64Disallow: /INSTALL.pgsql.txt
65Disallow: /INSTALL.sqlite.txt
66Disallow: /install.php
67Disallow: /INSTALL.txt
68Disallow: /LICENSE.txt
69Disallow: /MAINTAINERS.txt
70Disallow: /update.php
71Disallow: /UPGRADE.txt
72Disallow: /xmlrpc.php
73# Paths (clean URLs)
74Disallow: /admin
75Disallow: /comment/reply
76Disallow: /filter/tips
77Disallow: /node/add
78Disallow: /search
79Disallow: /user/register
80Disallow: /user/password
81Disallow: /user/login
82Disallow: /user/logout
83# Paths (no clean URLs)
84Disallow: /?q=admin
85Disallow: /?q=comment/reply
86Disallow: /?q=filter/tips
87Disallow: /?q=node/add
88Disallow: /?q=search
89Disallow: /?q=user/password
90Disallow: /?q=user/register
91Disallow: /?q=user/login
92Disallow: /?q=user/logout
93
94Sitemap: https://www.regeneron.com/sitemap.xml
95Sitemap: https://www.regeneron.com/images-sitemap.xml
96Sitemap: https://www.regeneron.com/video-sitemap.xml
97
98# End robots.txt