NetApp / robots.txt
← back to netapp.com · change detected 30d ago (2026-10-07T15:53:57Z) · modified +82 −27
What changed
- 🤖 AIAdded rules for AI crawler Amazonbot (Amazon)
- 🤖 AIAdded rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AIAdded rules for AI crawler CCBot (Common Crawl)
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler Claude-Web (Anthropic)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler cohere-ai (Cohere)
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- Removed Disallow: /as/ for *
- Removed Disallow: /au/ for *
- Removed Disallow: /br/ for *
- Removed Disallow: /ch/ for *
- Removed Disallow: /cn/ for *
- Removed Disallow: /il/ for *
- Removed Disallow: /jp/ for *
- Removed Disallow: /kr/ for *
- Removed Disallow: /mx/ for *
- Removed Disallow: /nl/ for *
- Removed Disallow: /ru/ for *
- Removed Disallow: /se/ for *
- Removed Disallow: /uk/ for *
- Removed Disallow: /us-en/ for *
- Removed Disallow: /us/ for *
- Added sitemap https://www.netapp.com/de/pdfsitemap.xml
- Added sitemap https://www.netapp.com/es/pdfsitemap.xml
- Added sitemap https://www.netapp.com/fr/pdfsitemap.xml
- Added sitemap https://www.netapp.com/it/pdfsitemap.xml
- Added sitemap https://www.netapp.com/ja/pdfsitemap.xml
- Added sitemap https://www.netapp.com/ko/pdfsitemap.xml
- Added sitemap https://www.netapp.com/pdfsitemap.xml
- Added sitemap https://www.netapp.com/pt/pdfsitemap.xml
- Added sitemap https://www.netapp.com/sitemap-video.xml
- Added sitemap https://www.netapp.com/zh-hans/pdfsitemap.xml
- Added sitemap https://www.netapp.com/zh-hant/pdfsitemap.xml
Diff old (2026-06-20T01:10:30Z) → new (2026-10-07T15:53:57Z)
| @@ -1,16 +1,12 @@ | ||
| 1 | + # ============================================ | |
| 2 | + # robots.txt - NetApp.com | |
| 3 | + # Last Updated: June 2026 | |
| 4 | + # ============================================ | |
| 5 | + | |
| 6 | + # ------------------------------------ | |
| 7 | + # General Crawlers | |
| 8 | + # ------------------------------------ | |
| 1 | 9 | User-agent: * |
| 2 | - | |
| 3 | - Sitemap: https://www.netapp.com/sitemap.xml | |
| 4 | - Sitemap: https://www.netapp.com/de/sitemap.xml | |
| 5 | - Sitemap: https://www.netapp.com/es/sitemap.xml | |
| 6 | - Sitemap: https://www.netapp.com/fr/sitemap.xml | |
| 7 | - Sitemap: https://www.netapp.com/it/sitemap.xml | |
| 8 | - Sitemap: https://www.netapp.com/ja/sitemap.xml | |
| 9 | - Sitemap: https://www.netapp.com/ko/sitemap.xml | |
| 10 | - Sitemap: https://www.netapp.com/pt/sitemap.xml | |
| 11 | - Sitemap: https://www.netapp.com/zh-hans/sitemap.xml | |
| 12 | - Sitemap: https://www.netapp.com/zh-hant/sitemap.xml | |
| 13 | - | |
| 14 | 10 | Disallow: /api/ |
| 15 | 11 | Disallow: /secure_document_area/ |
| 16 | 12 | Disallow: /welcome/ |
| @@ -25,18 +21,77 @@ | ||
| 25 | 21 | Disallow: /ru/welcome/ |
| 26 | 22 | Disallow: /zh-hans/welcome/ |
| 27 | 23 | Disallow: /zh-hant/welcome/ |
| 28 | - Disallow: /as/ | |
| 29 | - Disallow: /au/ | |
| 30 | - Disallow: /br/ | |
| 31 | - Disallow: /cn/ | |
| 32 | - Disallow: /jp/ | |
| 33 | - Disallow: /uk/ | |
| 34 | - Disallow: /us/ | |
| 35 | - Disallow: /kr/ | |
| 36 | - Disallow: /mx/ | |
| 37 | - Disallow: /nl/ | |
| 38 | - Disallow: /ru/ | |
| 39 | - Disallow: /se/ | |
| 40 | - Disallow: /il/ | |
| 41 | - Disallow: /us-en/ | |
| 42 | - Disallow: /ch/ | |
| 24 | + | |
| 25 | + # ------------------------------------ | |
| 26 | + # AI / LLM Crawlers - Explicitly Allowed | |
| 27 | + # ------------------------------------ | |
| 28 | + User-agent: GPTBot | |
| 29 | + Allow: / | |
| 30 | + | |
| 31 | + User-agent: ChatGPT-User | |
| 32 | + Allow: / | |
| 33 | + | |
| 34 | + User-agent: Claude-Web | |
| 35 | + Allow: / | |
| 36 | + | |
| 37 | + User-agent: ClaudeBot | |
| 38 | + Allow: / | |
| 39 | + | |
| 40 | + User-agent: PerplexityBot | |
| 41 | + Allow: / | |
| 42 | + | |
| 43 | + User-agent: Amazonbot | |
| 44 | + Allow: / | |
| 45 | + | |
| 46 | + User-agent: cohere-ai | |
| 47 | + Allow: / | |
| 48 | + | |
| 49 | + User-agent: anthropic-ai | |
| 50 | + Allow: / | |
| 51 | + | |
| 52 | + User-agent: CCBot | |
| 53 | + Allow: / | |
| 54 | + | |
| 55 | + User-agent: Google-Extended | |
| 56 | + Allow: / | |
| 57 | + | |
| 58 | + # ------------------------------------ | |
| 59 | + # Crawl Rate Management | |
| 60 | + # ------------------------------------ | |
| 61 | + Crawl-delay: 2 | |
| 62 | + | |
| 63 | + # ------------------------------------ | |
| 64 | + # Sitemaps | |
| 65 | + # ------------------------------------ | |
| 66 | + | |
| 67 | + # Primary Sitemaps | |
| 68 | + Sitemap: https://www.netapp.com/sitemap.xml | |
| 69 | + Sitemap: https://www.netapp.com/de/sitemap.xml | |
| 70 | + Sitemap: https://www.netapp.com/es/sitemap.xml | |
| 71 | + Sitemap: https://www.netapp.com/fr/sitemap.xml | |
| 72 | + Sitemap: https://www.netapp.com/it/sitemap.xml | |
| 73 | + Sitemap: https://www.netapp.com/ja/sitemap.xml | |
| 74 | + Sitemap: https://www.netapp.com/ko/sitemap.xml | |
| 75 | + Sitemap: https://www.netapp.com/pt/sitemap.xml | |
| 76 | + Sitemap: https://www.netapp.com/zh-hans/sitemap.xml | |
| 77 | + Sitemap: https://www.netapp.com/zh-hant/sitemap.xml | |
| 78 | + | |
| 79 | + # Video Sitemap (EN only) | |
| 80 | + Sitemap: https://www.netapp.com/sitemap-video.xml | |
| 81 | + | |
| 82 | + # PDF Sitemaps | |
| 83 | + Sitemap: https://www.netapp.com/pdfsitemap.xml | |
| 84 | + Sitemap: https://www.netapp.com/de/pdfsitemap.xml | |
| 85 | + Sitemap: https://www.netapp.com/es/pdfsitemap.xml | |
| 86 | + Sitemap: https://www.netapp.com/fr/pdfsitemap.xml | |
| 87 | + Sitemap: https://www.netapp.com/it/pdfsitemap.xml | |
| 88 | + Sitemap: https://www.netapp.com/pt/pdfsitemap.xml | |
| 89 | + Sitemap: https://www.netapp.com/ja/pdfsitemap.xml | |
| 90 | + Sitemap: https://www.netapp.com/ko/pdfsitemap.xml | |
| 91 | + Sitemap: https://www.netapp.com/zh-hans/pdfsitemap.xml | |
| 92 | + Sitemap: https://www.netapp.com/zh-hant/pdfsitemap.xml | |
| 93 | + | |
| 94 | + # ------------------------------------ | |
| 95 | + # LLM Content Reference | |
| 96 | + # ------------------------------------ | |
| 97 | + # For AI/LLM structured content, see: https://www.netapp.com/llms.txt |