Regeneron Pharmaceuticals / robots.txt
← back to regeneron.com · change detected 9d ago (2026-06-17T02:02:49Z) · modified +92 −1529
What changed
- Response is a valid robots.txt again (was an HTML page)
- Added rules for user-agent ‘*’
- 🤖 AIAdded rules for AI crawler Amazonbot (Amazon)
- 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
- 🤖 AIAdded rules for AI crawler Bytespider (ByteDance)
- 🤖 AIAdded rules for AI crawler CCBot (Common Crawl)
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Diffbot (Diffbot)
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler meta-externalagent (Meta)
- 🤖 AIAdded rules for AI crawler meta-externalfetcher (Meta)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- Added sitemap https://www.regeneron.com/images-sitemap.xml
- Added sitemap https://www.regeneron.com/sitemap.xml
- Added sitemap https://www.regeneron.com/video-sitemap.xml
Diff (no previous version) → (file removed)
No line-level differences (identical or both empty).