Nvidia / robots.txt
← back to nvidia.com · change detected 1mo ago (2026-08-12T09:19:38Z) · modified +18 −69
What changed
- 🤖 AIRemoved rules for AI crawler AI2Bot (Allen Institute for AI)
- Removed all rules for user-agent ‘Amazon-kendra’
- 🤖 AIRemoved rules for AI crawler Amazonbot (Amazon)
- 🤖 AIRemoved rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AIRemoved rules for AI crawler Applebot-Extended (Apple)
- Removed all rules for user-agent ‘AutoGPT’
- Removed all rules for user-agent ‘BedrockBot’
- Removed all rules for user-agent ‘Bingbot’
- 🤖 AIRemoved rules for AI crawler Bytespider (ByteDance)
- 🤖 AIRemoved rules for AI crawler CCBot (Common Crawl)
- Removed all rules for user-agent ‘Chatglm-spider’
- Removed all rules for user-agent ‘ChatGPT’
- 🤖 AIRemoved rules for AI crawler ChatGPT-User (OpenAI)
- Removed all rules for user-agent ‘Claude-search’
- 🤖 AIRemoved rules for AI crawler Claude-User (Anthropic)
- 🤖 AIRemoved rules for AI crawler Claude-Web (Anthropic)
- 🤖 AIRemoved rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIRemoved rules for AI crawler cohere-ai (Cohere)
- Removed all rules for user-agent ‘Cotoyogi’
- Removed all rules for user-agent ‘Deepseek’
- Removed all rules for user-agent ‘Deepseek Chat’
- Removed all rules for user-agent ‘Deepseekbot’
- 🤖 AIRemoved rules for AI crawler DuckAssistBot (DuckDuckGo)
- Removed all rules for user-agent ‘DuckDuckBot’
- 🤖 AIRemoved rules for AI crawler FacebookBot (Meta)
- Removed all rules for user-agent ‘Geminiios’
- Removed all rules for user-agent ‘Google AdsBot’
- 🤖 AIRemoved rules for AI crawler Google-Extended (Google)
- Removed all rules for user-agent ‘Googleagent-manier’
- Removed all rules for user-agent ‘Googlebot’
- Removed all rules for user-agent ‘Googlebot Smartphone’
- 🤖 AIRemoved rules for AI crawler GPTBot (OpenAI)
- Removed all rules for user-agent ‘Grok’
- Removed all rules for user-agent ‘ia-archiver’
- Removed all rules for user-agent ‘iaskbot’
- Removed all rules for user-agent ‘iaskspider’
- Removed all rules for user-agent ‘ICC-crawler’
- Removed all rules for user-agent ‘Linerbot’
- 🤖 AIRemoved rules for AI crawler meta-externalagent (Meta)
- 🤖 AIRemoved rules for AI crawler meta-externalfetcher (Meta)
- Removed all rules for user-agent ‘Meta-llama’
- Removed all rules for user-agent ‘Metaaib’
- 🤖 AIRemoved rules for AI crawler MistralAI-User (Mistral AI)
- Removed all rules for user-agent ‘NovaAct’
- 🤖 AIRemoved rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIRemoved rules for AI crawler PanguBot (Huawei)
- Removed all rules for user-agent ‘Perplexity’
- 🤖 AIRemoved rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIRemoved rules for AI crawler PerplexityBot (Perplexity)
- Removed all rules for user-agent ‘Phindbot’
- ... and 12 more change(s)
Diff old (2026-06-20T01:10:30Z) → new (2026-08-12T09:19:38Z)
| @@ -1,12 +1,12 @@ | ||
| 1 | 1 | # |
| 2 | - # Welcome to NVIDIA | |
| 2 | + # Welcome to NVIDIA | |
| 3 | 3 | # |
| 4 | 4 | #@@@@@@@@@@@@@@@@@@@@@@@@@@@///////////////////////////////////@@@@@@@@@@@@@@@@@@ |
| 5 | 5 | #@@@@@@@@@@@@@@@@@@@@@@@@@@@///////////////////////////////////@@@@@@@@@@@@@@@@@@ |
| 6 | 6 | #@@@@@@@@@@@@@@@@@@@@@@@@@@%#@#////////////////////////////////@@@@@@@@@@@@@@@@@@ |
| 7 | 7 | #@@@@@@@@@@@@@@@@@//////////@@@@@@@@@@@@///////////////////////@@@@@@@@@@@@@@@@@@ |
| 8 | 8 | #@@@@@@@@@@@@@//////@@@@@@@@/////////@@@@@@@///////////////////@@@@@@@@@@@@@@@@@@ |
| 9 | - #@@@@@@@@@//////@@@@@@//////@@@@@@@//////@@@@@@////////////////@@@@@@@@@@@@@@@@@@ | |
| 9 | + #@@@@@@@@@@@//////@@@@@@//////@@@@@@@//////@@@@@@////////////////@@@@@@@@@@@@@@@@@@ | |
| 10 | 10 | #@@@@@@//////@@@@@//////@@@@///@@@@@@@/////&@@@@@//////////////@@@@@@@@@@@@@@@@@@ |
| 11 | 11 | #@@@@@@//////@@@@////@@@@@@@/////@@@@/////@@@@@@///////////////@@@@@@@@@@@@@@@@@@ |
| 12 | 12 | #@@@@@@@@/////@@@@/////@@@@@////////////@@@@@///////@//////////@@@@@@@@@@@@@@@@@@ |
| @@ -26,76 +26,21 @@ | ||
| 26 | 26 | # Or check out our YouTube channel for our latest |
| 27 | 27 | # https://www.youtube.com/user/nvidia |
| 28 | 28 | # |
| 29 | - # Last updated 19 Nov 2025 by LC | |
| 29 | + # Last updated 11 Aug 2026 by RR | |
| 30 | + # | |
| 30 | 31 | |
| 31 | - sitemap: https://www.nvidia.com/sitemap_index.xml | |
| 32 | + Sitemap: https://www.nvidia.com/sitemap_index.xml | |
| 33 | + LLMs.txt: https://www.nvidia.com/llms.txt | |
| 32 | 34 | |
| 33 | - # AI Agents | |
| 34 | - User-agent: ai2bot | |
| 35 | - User-agent: anthropic-ai | |
| 36 | - User-agent: Amazonbot | |
| 37 | - User-agent: Amazon-kendra | |
| 38 | - User-agent: Applebot-extended | |
| 39 | - User-agent: AutoGPT | |
| 40 | - User-agent: BedrockBot | |
| 41 | - User-agent: Bytespider | |
| 42 | - User-agent: CCBot | |
| 43 | - User-agent: Chatglm-spider | |
| 44 | - User-agent: ChatGPT | |
| 45 | - User-agent: ChatGPT-User | |
| 46 | - User-agent: Claudebot | |
| 47 | - User-agent: Claude-user | |
| 48 | - User-agent: Claude-search | |
| 49 | - User-agent: Claude-web | |
| 50 | - User-agent: Cohere-ai | |
| 51 | - User-agent: Cotoyogi | |
| 52 | - User-agent: Deepseek | |
| 53 | - User-agent: Deepseek Chat | |
| 54 | - User-agent: Deepseekbot | |
| 55 | - User-agent: DuckAssistBot | |
| 56 | - User-agent: DuckDuckBot | |
| 57 | - User-agent: FacebookBot | |
| 58 | - User-agent: Geminiios | |
| 59 | - User-agent: Google-Extended | |
| 60 | - User-agent: Googleagent-manier | |
| 61 | - User-agent: GPTBot | |
| 62 | - User-agent: Grok | |
| 63 | - User-agent: ia-archiver | |
| 64 | - User-agent: iaskbot | |
| 65 | - User-agent: iaskspider | |
| 66 | - User-agent: ICC-crawler | |
| 67 | - User-agent: Linerbot | |
| 68 | - User-agent: Metaaib | |
| 69 | - User-agent: Meta-externalagent | |
| 70 | - User-agent: Meta-externalfetcher | |
| 71 | - User-agent: Meta-llama | |
| 72 | - User-agent: Mistralai-user | |
| 73 | - User-agent: NovaAct | |
| 74 | - User-agent: OAI-Searchbot | |
| 75 | - User-agent: Pangubot | |
| 76 | - User-agent: Perplexity | |
| 77 | - User-agent: PerplexityBot | |
| 78 | - User-agent: Perplexity-user | |
| 79 | - User-agent: Phindbot | |
| 80 | - User-agent: PoeBot | |
| 81 | - User-agent: Qwen | |
| 82 | - User-agent: Sogou | |
| 83 | - User-agent: Timpibot | |
| 84 | - User-agent: Tongyi | |
| 85 | - User-agent: Wenxiaobai | |
| 86 | - User-agent: Xinghuo | |
| 87 | - User-agent: Youbot | |
| 88 | - User-agent: Yuanbaobot | |
| 89 | - Allow: /*.llms.txt$ | |
| 90 | - Allow: /*.md$ | |
| 91 | - Allow: https://www.nvidia.com/llms.txt | |
| 92 | 35 | |
| 93 | - User-agent: Googlebot | |
| 94 | - User-agent: Googlebot Smartphone | |
| 95 | - User-agent: Google AdsBot | |
| 96 | - User-agent: Bingbot | |
| 97 | - Disallow: /*.md$ | |
| 98 | - Disallow:/*.llms.txt$ | |
| 36 | + # AI / LLM crawlers & fetchers | |
| 37 | + | |
| 38 | + User-agent: * | |
| 39 | + Content-Signal: ai-train=yes, search=yes, ai-input=yes | |
| 40 | + Allow: / | |
| 41 | + | |
| 42 | + | |
| 43 | + # Disallow Rules | |
| 99 | 44 | |
| 100 | 45 | User-agent: * |
| 101 | 46 | |
| @@ -111,6 +56,10 @@ | ||
| 111 | 56 | Disallow: /*&topicPage= |
| 112 | 57 | Disallow: /*?commentPage= |
| 113 | 58 | Disallow: /*&commentPage= |
| 59 | + | |
| 60 | + # Account & Authentication | |
| 61 | + Disallow: /*/training/academy/auth/ | |
| 62 | + Disallow: /*/training/academy/login/ | |
| 114 | 63 | |
| 115 | 64 | Disallow: /admin/ |
| 116 | 65 | Disallow: /attach/ |