Fortra / robots.txt
← back to fortra.com · change detected 20h ago (2026-09-18T15:27:37Z) · modified +7 −0
What changed
- Added rules for user-agent ‘SemrushBot’
- Added rules for user-agent ‘SiteAuditBot’
Diff old (2026-08-06T19:27:21Z) → new (2026-09-18T15:27:37Z)
| @@ -99,6 +99,13 @@ | ||
| 99 | 99 | User-agent: TikTokSpider |
| 100 | 100 | Allow: / |
| 101 | 101 | |
| 102 | + # SEMrush | |
| 103 | + User-agent: SiteAuditBot | |
| 104 | + Crawl-delay: 5 | |
| 105 | + | |
| 106 | + User-agent: SemrushBot | |
| 107 | + Crawl-delay: 5 | |
| 108 | + | |
| 102 | 109 | # Fallback |
| 103 | 110 | User-agent: * |
| 104 | 111 | Allow: / |