Cisco Systems / robots.txt
← back to cisco.com · change detected 21d ago (2026-10-16T19:00:39Z) · modified +0 −19
What changed
- Removed all rules for user-agent ‘Bingbot’
- 🤖 AIRemoved rules for AI crawler CCBot (Common Crawl)
- 🤖 AIRemoved rules for AI crawler ClaudeBot (Anthropic)
- Removed all rules for user-agent ‘Google AdsBot’
- 🤖 AIRemoved rules for AI crawler Google-Extended (Google)
- Removed all rules for user-agent ‘Googlebot’
- Removed all rules for user-agent ‘Googlebot Smartphone’
- 🤖 AIRemoved rules for AI crawler GPTBot (OpenAI)
- Removed sitemap https://www.cisco.com/web/sitemap/www_cisco_com_migration_at_scale_index.xml
- Removed sitemap https://www.cisco.com/web/sitemap/www_cisco_com_migration_at_scale_on_prem_index.xml
Diff old (2026-06-20T01:10:30Z) → new (2026-10-16T19:00:39Z)
| @@ -6,23 +6,6 @@ | ||
| 6 | 6 | Disallow: /univ-src/ccden # will get content through /univercd |
| 7 | 7 | Disallow: /cpropub/univercd # obsolete |
| 8 | 8 | Disallow: /jobs # temporary entry per performance team |
| 9 | - #-------------------------------- | |
| 10 | - User-agent: GPTBot | |
| 11 | - User-agent: CCBot | |
| 12 | - User-agent: Google-Extended | |
| 13 | - User-agent: ClaudeBot | |
| 14 | - Allow: /site/us/en/ | |
| 15 | - Allow: /c/us/en/ | |
| 16 | - Allow: /*.llms.txt$ | |
| 17 | - Allow: /*.md$ | |
| 18 | - Allow: https://www.cisco.com/llms.txt | |
| 19 | - | |
| 20 | - User-agent: Googlebot | |
| 21 | - User-agent: Googlebot Smartphone | |
| 22 | - User-agent: Google AdsBot | |
| 23 | - User-agent: Bingbot | |
| 24 | - Disallow: /*.md$ | |
| 25 | - Disallow:/*.llms.txt$ | |
| 26 | 9 | #-------------------------------- |
| 27 | 10 | User-agent: * |
| 28 | 11 | Disallow: /*.html?cachemode=refresh$ |
| @@ -291,8 +274,6 @@ | ||
| 291 | 274 | Sitemap: https://www.cisco.com/web/sitemap/www_cisco_com_zh_cn_index.xml |
| 292 | 275 | Sitemap: https://www.cisco.com/web/sitemap/www_cisco_com_zh_hk_index.xml |
| 293 | 276 | Sitemap: https://www.cisco.com/web/sitemap/www_cisco_com_zh_tw_index.xml |
| 294 | - Sitemap: https://www.cisco.com/web/sitemap/www_cisco_com_migration_at_scale_on_prem_index.xml | |
| 295 | - Sitemap: https://www.cisco.com/web/sitemap/www_cisco_com_migration_at_scale_index.xml | |
| 296 | 277 | Sitemap: https://www.cisco.com/web/sitemap/rss/rss.xml |
| 297 | 278 | Sitemap: https://www.cisco.com/web/sitemap/rss/rss1.xml |
| 298 | 279 | Sitemap: https://www.cisco.com/web/sitemap/rss/rss2.xml |