Regeneron Pharmaceuticals / robots.txt

← back to regeneron.com · change detected 3d ago (2026-11-04T13:32:12Z) · modified +24 −1

What changed

		@@ -1,4 +1,23 @@
1		- #updated 05/14/2026
	1	+ # robots.txt — www.regeneron.com
	2	+ # updated 09/11/2026
	3	+ #
	4	+ # CHANGES IN THIS VERSION (vs. 05/14/2026):
	5	+ # 1. Added Content-Signal directives to BOTH crawler groups (see note below).
	6	+ # 2. No other rules added, removed, or reordered.
	7	+ #
	8	+ # NOTE ON CONTENT-SIGNAL PLACEMENT:
	9	+ # robots.txt group matching is most-specific-wins. A crawler named in the
	10	+ # "AI / LLM Crawlers" group below does NOT read the "User-agent: *" group at
	11	+ # all. Content-Signal must therefore appear in BOTH groups to reach every
	12	+ # crawler. Declaring it only once under "User-agent: *" would leave all 13
	13	+ # named AI agents without a signal.
	14	+ #
	15	+ # Declared preference: search=yes, ai-input=yes, ai-train=no
	16	+ # search=yes -> permit traditional search indexing
	17	+ # ai-input=yes -> permit use as grounding input for AI answers (citation)
	18	+ # ai-train=no -> decline use as generative model training data
	19	+ #
	20	+ # Reference: https://contentsignals.org
2	21
3	22	# AI / LLM Crawlers
4	23	User-agent: GPTBot
		@@ -14,6 +33,8 @@
14	33	User-agent: Applebot-Extended
15	34	User-agent: CCBot
16	35	User-agent: Diffbot
	36	+
	37	+ Content-Signal: search=yes, ai-input=yes, ai-train=no
17	38
18	39	Disallow: /about/leadership$
19	40	Disallow: /about/leadership/$
		@@ -54,6 +75,8 @@
54	75	Disallow: /?q=user/logout
55	76
56	77	User-agent: *
	78	+
	79	+ Content-Signal: search=yes, ai-input=yes, ai-train=no
57	80
58	81	Disallow: /node/*
59	82	Disallow: /search-results*