Humana / robots.txt
← back to humana.com · change detected 29d ago (2026-10-09T04:09:04Z) · modified +21 −0
What changed
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- Added rules for user-agent ‘Googlebot’
- Added rules for user-agent ‘GoogleOther’
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
Diff old (2026-06-20T01:10:30Z) → new (2026-10-09T04:09:04Z)
| @@ -1,6 +1,7 @@ | ||
| 1 | 1 | # Robots.txt file created by https://www.humana.com/ |
| 2 | 2 | # For domain: https://www.humana.com/ |
| 3 | 3 | # Updated 09-04-2023 - September 2023 release |
| 4 | + # Updated 10-05-2026 - explicit allow rules for verified AI/search crawler bots (CRC approved, see S0574) | |
| 4 | 5 | # All robots will spider the domain |
| 5 | 6 | |
| 6 | 7 | User-agent: * |
| @@ -10,4 +11,24 @@ | ||
| 10 | 11 | Disallow: /mvc/ |
| 11 | 12 | Disallow: /sitecore/ |
| 12 | 13 | |
| 14 | + # Network-level identity verification for the bots below (CIDR/reverse-DNS match) | |
| 15 | + # is enforced at the Akamai/Shape WAF layer, not by this file. | |
| 16 | + # OpenAI ChatGPT Search - CIDR verification feed: https://openai.com/searchbot.json | |
| 17 | + User-agent: OAI-SearchBot | |
| 18 | + Allow: / | |
| 19 | + | |
| 20 | + # Anthropic Claude - CIDR verification feed: https://claude.com/crawling/bots.json | |
| 21 | + User-agent: Claude-SearchBot | |
| 22 | + Allow: / | |
| 23 | + | |
| 24 | + User-agent: ClaudeBot | |
| 25 | + Allow: / | |
| 26 | + | |
| 27 | + # Google - verified via common-crawlers.json/special-crawlers.json or reverse DNS (googlebot.com/googleusercontent.com/google.com) | |
| 28 | + User-agent: Googlebot | |
| 29 | + Allow: / | |
| 30 | + | |
| 31 | + User-agent: GoogleOther | |
| 32 | + Allow: / | |
| 33 | + | |
| 13 | 34 | Sitemap: https://www.humana.com/sitemap.xml |