seoClarity / robots.txt
← back to seoclarity.net · change detected 22d ago (2026-10-15T18:35:45Z) · modified +19 −2
What changed
- Blocked user-agent ‘BrightEdge Crawler/1.0’ entirely
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- Added Disallow: /hubfs/Client%20Stories/Client%20Testimonials.pdf for *
- Added Disallow: /seo-platform-essentials/confirmation-lead/thank-you for *
- Removed Allow: /seo-platform-essentials/confirmation-lead/thank-you for *
- Removed Disallow: /workflow-video-library for *
- Added sitemap https://www.seoclarity.net/sitemap.xml
Diff old (2026-09-02T22:25:58Z) → new (2026-10-15T18:35:45Z)
| @@ -1,12 +1,29 @@ | ||
| 1 | + User-agent: BrightEdge Crawler/1.0 | |
| 2 | + Disallow: / | |
| 3 | + Disallow: /_hcms/preview/ | |
| 4 | + Disallow: /hs/manage-preferences/ | |
| 5 | + Disallow: /hs/preferences-center/ | |
| 6 | + Disallow: /*?*hs_preview=* | |
| 7 | + Disallow: /*?*hsCacheBuster=* | |
| 8 | + | |
| 9 | + User-agent: OAI-SearchBot | |
| 10 | + Allow: / | |
| 11 | + Disallow: /_hcms/preview/ | |
| 12 | + Disallow: /hs/manage-preferences/ | |
| 13 | + Disallow: /hs/preferences-center/ | |
| 14 | + Disallow: /*?*hs_preview=* | |
| 15 | + Disallow: /*?*hsCacheBuster=* | |
| 16 | + | |
| 1 | 17 | User-agent: * |
| 2 | 18 | Disallow: /thank-you |
| 3 | 19 | Disallow: /thank-you/ |
| 4 | 20 | Disallow: /thank-you* |
| 5 | 21 | Disallow: /*thank-you$ |
| 6 | - Allow: /seo-platform-essentials/confirmation-lead/thank-you | |
| 22 | + Disallow: /seo-platform-essentials/confirmation-lead/thank-you | |
| 7 | 23 | Disallow: /competitive-insights-report-3/ |
| 8 | 24 | Disallow: /see-it-in-action-enterprise-v1 |
| 9 | - Disallow: /workflow-video-library | |
| 25 | + Disallow: /hubfs/Client%20Stories/Client%20Testimonials.pdf | |
| 26 | + Sitemap: https://www.seoclarity.net/sitemap.xml | |
| 10 | 27 | Disallow: /_hcms/preview/ |
| 11 | 28 | Disallow: /hs/manage-preferences/ |
| 12 | 29 | Disallow: /hs/preferences-center/ |