NeuralCrawl

Northrop Grumman / robots.txt snapshot

← back to northropgrumman.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 828 bytes · sha256 3d058a941886d17e · raw

final URL: https://www.northropgrumman.com/robots.txt

1# ALLOW AI SEARCH & REFERRAL BOTS
2# These bots browse the web to answer user queries with links/citations.
3User-agent: OAI-SearchBot
4User-agent: ChatGPT-User
5User-agent: PerplexityBot
6User-agent: Claude-Web
7User-agent: YouBot
8User-agent: Bingbot
9Allow: /
10
11# DISALLOW AI TRAINING & MODEL CRAWLERS
12# These bots scrape data for model training.
13User-agent: GPTBot
14User-agent: ClaudeBot
15User-agent: Google-Extended
16User-agent: Applebot-Extended
17User-agent: CCBot
18User-agent: FacebookBot
19User-agent: Amazonbot
20User-agent: Cohere-ai
21User-agent: Omgilibot
22User-agent: Omgili
23Disallow: /
24
25# GENERAL SEARCH ENGINE RULES
26User-agent: *
27Allow: /
28Disallow: /admin/
29Disallow: /wp-admin/
30Disallow: /?s=
31Disallow: /api/
32
33# SITEMAPS
34Sitemap: https://www.northropgrumman.com/videos-sitemap.xml
35Sitemap: https://www.northropgrumman.com/sitemap.xml