General Dynamics / robots.txt
← back to gd.com · change detected 9h ago (2026-11-09T15:07:03Z) · modified +64 −0
What changed
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘AdsBot-Google’
- Added rules for user-agent ‘Applebot’
- Added rules for user-agent ‘archive.org_bot’
- Added rules for user-agent ‘bingbot’
- Added rules for user-agent ‘BingPreview’
- Added rules for user-agent ‘DeepCrawl’
- Added rules for user-agent ‘Googlebot’
- Added rules for user-agent ‘Googlebot-Image’
- Added rules for user-agent ‘Googlebot-Mobile’
- Added rules for user-agent ‘ia_archiver’
- Added rules for user-agent ‘Mediapartners-Google’
- Added rules for user-agent ‘msnbot’
- Added rules for user-agent ‘ScoutJet’
- Added rules for user-agent ‘SearchmetricsBot’
- Added rules for user-agent ‘SeznamBot’
- Added rules for user-agent ‘STC-bot’
- Added rules for user-agent ‘TelegramBot’
- Added rules for user-agent ‘Twitterbot’
- Added rules for user-agent ‘Yahoo! Slurp’
- Added rules for user-agent ‘Yandex’
- Added sitemap https://gd.com/sitemap.xml
Diff old (2026-06-20T01:10:30Z) → new (2026-11-09T15:07:03Z)
| @@ -1,0 +1,64 @@ | ||
| 1 | + User-agent: AdsBot-Google | |
| 2 | + Disallow: /privacy-policy/ | |
| 3 | + | |
| 4 | + User-agent: Applebot | |
| 5 | + Disallow: /privacy-policy/ | |
| 6 | + | |
| 7 | + User-agent: BingPreview | |
| 8 | + Disallow: /privacy-policy/ | |
| 9 | + | |
| 10 | + User-agent: DeepCrawl | |
| 11 | + Disallow: /privacy-policy/ | |
| 12 | + | |
| 13 | + User-agent: Googlebot | |
| 14 | + Disallow: /privacy-policy/ | |
| 15 | + | |
| 16 | + User-agent: Googlebot-Image | |
| 17 | + Disallow: /privacy-policy/ | |
| 18 | + | |
| 19 | + User-agent: Googlebot-Mobile | |
| 20 | + Disallow: /privacy-policy/ | |
| 21 | + | |
| 22 | + User-agent: Mediapartners-Google | |
| 23 | + Disallow: /privacy-policy/ | |
| 24 | + | |
| 25 | + User-agent: STC-bot | |
| 26 | + Disallow: /privacy-policy/ | |
| 27 | + | |
| 28 | + User-agent: ScoutJet | |
| 29 | + Disallow: /privacy-policy/ | |
| 30 | + | |
| 31 | + User-agent: SearchmetricsBot | |
| 32 | + Disallow: /privacy-policy/ | |
| 33 | + | |
| 34 | + User-agent: SeznamBot | |
| 35 | + Disallow: /privacy-policy/ | |
| 36 | + | |
| 37 | + User-agent: TelegramBot | |
| 38 | + Disallow: /privacy-policy/ | |
| 39 | + | |
| 40 | + User-agent: Twitterbot | |
| 41 | + Disallow: /privacy-policy/ | |
| 42 | + | |
| 43 | + User-agent: Yahoo! Slurp | |
| 44 | + Disallow: /privacy-policy/ | |
| 45 | + | |
| 46 | + User-agent: Yandex | |
| 47 | + Disallow: /privacy-policy/ | |
| 48 | + | |
| 49 | + User-agent: archive.org_bot | |
| 50 | + Disallow: /privacy-policy/ | |
| 51 | + | |
| 52 | + User-agent: bingbot | |
| 53 | + Disallow: /privacy-policy/ | |
| 54 | + | |
| 55 | + User-agent: ia_archiver | |
| 56 | + Disallow: /privacy-policy/ | |
| 57 | + | |
| 58 | + User-agent: msnbot | |
| 59 | + Disallow: /privacy-policy/ | |
| 60 | + | |
| 61 | + User-agent: * | |
| 62 | + Disallow: | |
| 63 | + | |
| 64 | + Sitemap: https://gd.com/sitemap.xml |