NeuralCrawl

Huntington Ingalls Industries / robots.txt snapshot

← back to hii.com · fetched 2026-07-08T20:50:19Z (28d ago) · HTTP 200 · 1531 bytes · sha256 0b3cd2cb09a4f5fb · raw

final URL: https://www.hii.com/robots.txt

1# Default rules for all crawlers
2User-agent: *
3Allow: /
4Disallow: /api/
5Disallow: /_next/
6Disallow: /cms/
7
8# =============================================================
9# Explicitly allow AI crawlers
10# Why: Some crawlers check bot-specific rules first.
11# An explicit Allow signals intent — you WANT to be indexed.
12# =============================================================
13
14# OpenAI
15User-agent: GPTBot
16Allow: /
17
18User-agent: ChatGPT-User
19Allow: /
20
21User-agent: OAI-SearchBot
22Allow: /
23
24# Anthropic
25User-agent: ClaudeBot
26Allow: /
27
28User-agent: anthropic-ai
29Allow: /
30
31# Google AI
32User-agent: Google-Extended
33Allow: /
34
35User-agent: GoogleOther
36Allow: /
37
38# Perplexity
39User-agent: PerplexityBot
40Allow: /
41
42# You.com
43User-agent: YouBot
44Allow: /
45
46# Cohere
47User-agent: cohere-ai
48Allow: /
49
50# Apple
51User-agent: Applebot-Extended
52Allow: /
53
54# Microsoft / Bing
55User-agent: bingbot
56Allow: /
57
58# Meta
59User-agent: FacebookBot
60Allow: /
61
62# =============================================================
63# Host — Primary site
64# =============================================================
65Host: https://www.hii.com
66
67# =============================================================
68# Sitemap — Full crawlable URL structure
69# =============================================================
70Sitemap: https://www.hii.com/sitemap.xml
71
72# =============================================================
73# llms.txt — Token-efficient index of top level content
74# =============================================================
75llms.txt: https://www.hii.com/llms.txt