NeuralCrawl

Regeneron Pharmaceuticals / robots.txt snapshot

← back to regeneron.com · fetched 2026-11-04T13:32:12Z (3d ago) · HTTP 200 · 3270 bytes · sha256 d3dd2cda3e794c2f · raw

final URL: https://www.regeneron.com/robots.txt

1# robots.txt — www.regeneron.com
2# updated 09/11/2026
3#
4# CHANGES IN THIS VERSION (vs. 05/14/2026):
5# 1. Added Content-Signal directives to BOTH crawler groups (see note below).
6# 2. No other rules added, removed, or reordered.
7#
8# NOTE ON CONTENT-SIGNAL PLACEMENT:
9# robots.txt group matching is most-specific-wins. A crawler named in the
10# "AI / LLM Crawlers" group below does NOT read the "User-agent: *" group at
11# all. Content-Signal must therefore appear in BOTH groups to reach every
12# crawler. Declaring it only once under "User-agent: *" would leave all 13
13# named AI agents without a signal.
14#
15# Declared preference: search=yes, ai-input=yes, ai-train=no
16# search=yes -> permit traditional search indexing
17# ai-input=yes -> permit use as grounding input for AI answers (citation)
18# ai-train=no -> decline use as generative model training data
19#
20# Reference: https://contentsignals.org
21
22# AI / LLM Crawlers
23User-agent: GPTBot
24User-agent: ChatGPT-User
25User-agent: Google-Extended
26User-agent: ClaudeBot
27User-agent: PerplexityBot
28User-agent: Perplexity-User
29User-agent: Bytespider
30User-agent: Meta-ExternalAgent
31User-agent: Meta-ExternalFetcher
32User-agent: Amazonbot
33User-agent: Applebot-Extended
34User-agent: CCBot
35User-agent: Diffbot
36
37Content-Signal: search=yes, ai-input=yes, ai-train=no
38
39Disallow: /about/leadership$
40Disallow: /about/leadership/$
41Disallow: /node/*
42Disallow: /search-results*
43# Files
44Disallow: /CHANGELOG.txt
45Disallow: /cron.php
46Disallow: /INSTALL.mysql.txt
47Disallow: /INSTALL.pgsql.txt
48Disallow: /INSTALL.sqlite.txt
49Disallow: /install.php
50Disallow: /INSTALL.txt
51Disallow: /LICENSE.txt
52Disallow: /MAINTAINERS.txt
53Disallow: /update.php
54Disallow: /UPGRADE.txt
55Disallow: /xmlrpc.php
56# Paths (clean URLs)
57Disallow: /admin
58Disallow: /comment/reply
59Disallow: /filter/tips
60Disallow: /node/add
61Disallow: /search
62Disallow: /user/register
63Disallow: /user/password
64Disallow: /user/login
65Disallow: /user/logout
66# Paths (no clean URLs)
67Disallow: /?q=admin
68Disallow: /?q=comment/reply
69Disallow: /?q=filter/tips
70Disallow: /?q=node/add
71Disallow: /?q=search
72Disallow: /?q=user/password
73Disallow: /?q=user/register
74Disallow: /?q=user/login
75Disallow: /?q=user/logout
76
77User-agent: *
78
79Content-Signal: search=yes, ai-input=yes, ai-train=no
80
81Disallow: /node/*
82Disallow: /search-results*
83# Files
84Disallow: /CHANGELOG.txt
85Disallow: /cron.php
86Disallow: /INSTALL.mysql.txt
87Disallow: /INSTALL.pgsql.txt
88Disallow: /INSTALL.sqlite.txt
89Disallow: /install.php
90Disallow: /INSTALL.txt
91Disallow: /LICENSE.txt
92Disallow: /MAINTAINERS.txt
93Disallow: /update.php
94Disallow: /UPGRADE.txt
95Disallow: /xmlrpc.php
96# Paths (clean URLs)
97Disallow: /admin
98Disallow: /comment/reply
99Disallow: /filter/tips
100Disallow: /node/add
101Disallow: /search
102Disallow: /user/register
103Disallow: /user/password
104Disallow: /user/login
105Disallow: /user/logout
106# Paths (no clean URLs)
107Disallow: /?q=admin
108Disallow: /?q=comment/reply
109Disallow: /?q=filter/tips
110Disallow: /?q=node/add
111Disallow: /?q=search
112Disallow: /?q=user/password
113Disallow: /?q=user/register
114Disallow: /?q=user/login
115Disallow: /?q=user/logout
116
117Sitemap: https://www.regeneron.com/sitemap.xml
118Sitemap: https://www.regeneron.com/images-sitemap.xml
119Sitemap: https://www.regeneron.com/video-sitemap.xml
120
121# End robots.txt