SAP / robots.txt
← back to sap.com · change detected 9d ago (2026-09-09T12:27:15Z) · modified +45 −8
What changed
- 🤖 AIAdded Disallow: /fetch for Amazonbot
- 🤖 AIAdded Disallow: /proxy for Amazonbot
- 🤖 AIAdded Disallow: /sharedlibs/ for Amazonbot
- 🤖 AIAdded Disallow: /fetch for ChatGPT-User
- 🤖 AIAdded Disallow: /proxy for ChatGPT-User
- 🤖 AIAdded Disallow: /sharedlibs/ for ChatGPT-User
- 🤖 AIAdded Disallow: /fetch for Claude-SearchBot
- 🤖 AIAdded Disallow: /proxy for Claude-SearchBot
- 🤖 AIAdded Disallow: /sharedlibs/ for Claude-SearchBot
- 🤖 AIAdded Disallow: /fetch for Claude-User
- 🤖 AIAdded Disallow: /proxy for Claude-User
- 🤖 AIAdded Disallow: /sharedlibs/ for Claude-User
- 🤖 AIAdded Disallow: /fetch for ClaudeBot
- 🤖 AIAdded Disallow: /proxy for ClaudeBot
- 🤖 AIAdded Disallow: /sharedlibs/ for ClaudeBot
- 🤖 AIAdded Disallow: /fetch for GPTBot
- 🤖 AIAdded Disallow: /proxy for GPTBot
- 🤖 AIAdded Disallow: /sharedlibs/ for GPTBot
Diff old (2026-08-31T15:31:09Z) → new (2026-09-09T12:27:15Z)
| @@ -12,12 +12,12 @@ | ||
| 12 | 12 | # |
| 13 | 13 | # Find your next job: https://www.sap.com/careers |
| 14 | 14 | # |
| 15 | - # Last updated 2026-08-31 | |
| 15 | + # Last updated 2026-09-09 | |
| 16 | 16 | |
| 17 | 17 | User-agent: * |
| 18 | 18 | Disallow: / |
| 19 | 19 | |
| 20 | - # Search Engines & AI Agents (all share the same rule set) | |
| 20 | + # Search Engines (all share the same rule set) | |
| 21 | 21 | User-agent: Googlebot |
| 22 | 22 | User-agent: Googlebot-Image |
| 23 | 23 | User-agent: Googlebot-News |
| @@ -40,20 +40,57 @@ | ||
| 40 | 40 | User-agent: YandexAccessibilityBot |
| 41 | 41 | User-agent: YandexMobileBot |
| 42 | 42 | User-agent: Yandex |
| 43 | - User-agent: Amazonbot | |
| 44 | - User-agent: ClaudeBot | |
| 45 | - User-agent: Claude-User | |
| 46 | - User-agent: Claude-SearchBot | |
| 47 | 43 | User-agent: CCBot |
| 48 | 44 | User-agent: Google-Extended |
| 49 | 45 | User-agent: FacebookBot |
| 50 | 46 | User-agent: MistralAI-User |
| 51 | - User-agent: GPTBot | |
| 52 | - User-agent: ChatGPT-User | |
| 53 | 47 | User-agent: OAI-SearchBot |
| 54 | 48 | User-agent: PerplexityBot |
| 55 | 49 | User-agent: Perplexity-User |
| 56 | 50 | User-agent: archive.org_bot |
| 51 | + Disallow: /bin/sapdx/ | |
| 52 | + Disallow: /bin/sapdxc/ | |
| 53 | + Disallow: /design-system/digital/ | |
| 54 | + Disallow: /dmc/ | |
| 55 | + Disallow: /heartbeat.html | |
| 56 | + Disallow: /investors/de/stock/fixed-income/bond-documentation* | |
| 57 | + Disallow: /investors/de/stock/fixed-income/cp-documentation* | |
| 58 | + Disallow: /investors/en/stock/fixed-income/bond-documentation* | |
| 59 | + Disallow: /investors/en/stock/fixed-income/cp-documentation* | |
| 60 | + Disallow: */_jcr_content*.html | |
| 61 | + Disallow: */abm/* | |
| 62 | + Disallow: */cmp/dg/gdpr/* | |
| 63 | + Disallow: */cmp/dg/gdpr-emea/* | |
| 64 | + Disallow: */cmp/dg/gdpr-germany/* | |
| 65 | + Disallow: */cmp/eml/* | |
| 66 | + Disallow: */cmp/oth/crm-wx18-xrg-geb-fba/* | |
| 67 | + Disallow: */cmp/oth/crm-wx18-xrg-geb-supfba/* | |
| 68 | + Disallow: */cmp/oth/gdpr-opt-out/* | |
| 69 | + Disallow: */cmp/spm/* | |
| 70 | + Disallow: */cmp/td/sap-sustainability-data-exchange* | |
| 71 | + Disallow: */events/sapphire/orlando/celebration-night-concert.html | |
| 72 | + Disallow: */events/sponsorships.html | |
| 73 | + Disallow: */legal/video-surveillance* | |
| 74 | + Disallow: */products/erp/s4hana-private-edition/trial.html | |
| 75 | + Disallow: */tye.html | |
| 76 | + Disallow: */typ.html | |
| 77 | + Disallow: */registration/*? | |
| 78 | + Disallow: /silent.html | |
| 79 | + Disallow: /*/silent.html | |
| 80 | + Allow: /abm/dod.html | |
| 81 | + Allow: /abm/dod/* | |
| 82 | + Allow: /dmc/exp/* | |
| 83 | + | |
| 84 | + # AI Agents (all share the same rule set) | |
| 85 | + User-agent: ClaudeBot | |
| 86 | + User-agent: Claude-User | |
| 87 | + User-agent: Claude-SearchBot | |
| 88 | + User-agent: GPTBot | |
| 89 | + User-agent: ChatGPT-User | |
| 90 | + User-agent: Amazonbot | |
| 91 | + Disallow: /fetch | |
| 92 | + Disallow: /proxy | |
| 93 | + Disallow: /sharedlibs/ | |
| 57 | 94 | Disallow: /bin/sapdx/ |
| 58 | 95 | Disallow: /bin/sapdxc/ |
| 59 | 96 | Disallow: /design-system/digital/ |