Huntington Ingalls Industries / robots.txt
← back to hii.com · change detected 27d ago (2026-07-08T20:50:19Z) · modified +70 −3
What changed
- 🤖 AIAdded rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘bingbot’
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler cohere-ai (Cohere)
- 🤖 AIAdded rules for AI crawler FacebookBot (Meta)
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- Added rules for user-agent ‘GoogleOther’
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- 🤖 AIAdded rules for AI crawler YouBot (You.com)
- Added sitemap https://www.hii.com/sitemap.xml
- Removed sitemap https://hii.com/sitemap.xml
Diff old (2026-06-20T01:10:31Z) → new (2026-07-08T20:50:19Z)
| @@ -1,8 +1,75 @@ | ||
| 1 | - User-Agent: * | |
| 1 | + # Default rules for all crawlers | |
| 2 | + User-agent: * | |
| 2 | 3 | Allow: / |
| 3 | 4 | Disallow: /api/ |
| 4 | 5 | Disallow: /_next/ |
| 5 | 6 | Disallow: /cms/ |
| 6 | 7 | |
| 7 | - Host: https://hii.com | |
| 8 | - Sitemap: https://hii.com/sitemap.xml | |
| 8 | + # ============================================================= | |
| 9 | + # Explicitly allow AI crawlers | |
| 10 | + # Why: Some crawlers check bot-specific rules first. | |
| 11 | + # An explicit Allow signals intent — you WANT to be indexed. | |
| 12 | + # ============================================================= | |
| 13 | + | |
| 14 | + # OpenAI | |
| 15 | + User-agent: GPTBot | |
| 16 | + Allow: / | |
| 17 | + | |
| 18 | + User-agent: ChatGPT-User | |
| 19 | + Allow: / | |
| 20 | + | |
| 21 | + User-agent: OAI-SearchBot | |
| 22 | + Allow: / | |
| 23 | + | |
| 24 | + # Anthropic | |
| 25 | + User-agent: ClaudeBot | |
| 26 | + Allow: / | |
| 27 | + | |
| 28 | + User-agent: anthropic-ai | |
| 29 | + Allow: / | |
| 30 | + | |
| 31 | + # Google AI | |
| 32 | + User-agent: Google-Extended | |
| 33 | + Allow: / | |
| 34 | + | |
| 35 | + User-agent: GoogleOther | |
| 36 | + Allow: / | |
| 37 | + | |
| 38 | + # Perplexity | |
| 39 | + User-agent: PerplexityBot | |
| 40 | + Allow: / | |
| 41 | + | |
| 42 | + # You.com | |
| 43 | + User-agent: YouBot | |
| 44 | + Allow: / | |
| 45 | + | |
| 46 | + # Cohere | |
| 47 | + User-agent: cohere-ai | |
| 48 | + Allow: / | |
| 49 | + | |
| 50 | + # Apple | |
| 51 | + User-agent: Applebot-Extended | |
| 52 | + Allow: / | |
| 53 | + | |
| 54 | + # Microsoft / Bing | |
| 55 | + User-agent: bingbot | |
| 56 | + Allow: / | |
| 57 | + | |
| 58 | + # Meta | |
| 59 | + User-agent: FacebookBot | |
| 60 | + Allow: / | |
| 61 | + | |
| 62 | + # ============================================================= | |
| 63 | + # Host — Primary site | |
| 64 | + # ============================================================= | |
| 65 | + Host: https://www.hii.com | |
| 66 | + | |
| 67 | + # ============================================================= | |
| 68 | + # Sitemap — Full crawlable URL structure | |
| 69 | + # ============================================================= | |
| 70 | + Sitemap: https://www.hii.com/sitemap.xml | |
| 71 | + | |
| 72 | + # ============================================================= | |
| 73 | + # llms.txt — Token-efficient index of top level content | |
| 74 | + # ============================================================= | |
| 75 | + llms.txt: https://www.hii.com/llms.txt |