Varonis / robots.txt
← back to varonis.com · change detected 3d ago (2026-11-04T01:20:07Z) · modified +10 −5
What changed
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- Removed Disallow: /*?*hsCacheBuster=* for *
- Removed Disallow: /*?*hs_preview=* for *
- Removed Disallow: /_hcms/preview/ for *
- Removed Disallow: /hs/manage-preferences/ for *
- Removed Disallow: /hs/preferences-center/ for *
- Added sitemap https://varonis-mcp.webflow.io/sitemap.xml
- Added sitemap https://www.varonis.com/sitemap.xml
Diff old (2026-09-02T22:25:58Z) → new (2026-11-04T01:20:07Z)
| @@ -4,8 +4,13 @@ | ||
| 4 | 4 | Disallow: /en/state-of-data-security-2025-healthcare-life-sciences |
| 5 | 5 | Disallow: /service-unavailable |
| 6 | 6 | Disallow: /test-event-2026-attackers-playbook-2026-06-23 |
| 7 | - Disallow: /_hcms/preview/ | |
| 8 | - Disallow: /hs/manage-preferences/ | |
| 9 | - Disallow: /hs/preferences-center/ | |
| 10 | - Disallow: /*?*hs_preview=* | |
| 11 | - Disallow: /*?*hsCacheBuster=* | |
| 7 | + | |
| 8 | + User-agent: GPTbot | |
| 9 | + Allow: / | |
| 10 | + | |
| 11 | + User-agent: ClaudeBot | |
| 12 | + Allow: / | |
| 13 | + | |
| 14 | + Sitemap: https://www.varonis.com/sitemap.xml | |
| 15 | + | |
| 16 | + Sitemap: https://varonis-mcp.webflow.io/sitemap.xml |