Moz / robots.txt
← back to moz.com · change detected 11d ago (2026-09-08T18:06:18Z) · modified +4 −54
What changed
- Removed all rules for user-agent ‘AhrefsBot’
- Removed all rules for user-agent ‘DDM-DCipher’
- 🤖 AIRemoved rules for AI crawler GPTBot (OpenAI)
- Removed all rules for user-agent ‘rogerbot’
- Blocked user-agent ‘*’ entirely
- Removed Allow: /learn/ for *
- Removed Allow: /local/ for *
- Removed Allow: /products/ for *
- Removed Allow: /researchtools/ose/ for *
- Removed Allow: /researchtools/ose/$ for *
- Removed Allow: /researchtools/ose/anchors$ for *
- Removed Allow: /researchtools/ose/domains$ for *
- Removed Allow: /researchtools/ose/dotbot$ for *
- Removed Allow: /researchtools/ose/just-discovered$ for *
- Removed Allow: /researchtools/ose/links$ for *
- Removed Allow: /researchtools/ose/pages$ for *
- Removed Disallow: /admin-preview/* for *
- Removed Disallow: /api/user?* for *
- Removed Disallow: /checkout/* for *
- Removed Disallow: /community/q/questions/*/view_counts for *
- Removed Disallow: /community/q/socket.io/* for *
- Removed Disallow: /content/audit/* for *
- Removed Disallow: /content/search/* for *
- Removed Disallow: /cpresources/ for *
- Removed Disallow: /local/details/ for *
- Removed Disallow: /local/enterprise/confirm for *
- Removed Disallow: /local/search/ for *
- Removed Disallow: /marketplace/ for *
- Removed Disallow: /messages/ for *
- Removed Disallow: /page-strength/* for *
- Removed Disallow: /products/content/ for *
- Removed Disallow: /purchase/* for *
- Removed Disallow: /researchtools/ose/ for *
- Removed Disallow: /thumbs/* for *
- Removed Disallow: /vendor/ for *
- Removed sitemap https://moz.com/community/q/sitemap.xml
- Removed sitemap https://moz.com/sitemaps-1-sitemap.xml
Diff old (2026-06-20T14:56:27Z) → new (2026-09-08T18:06:18Z)
| @@ -1,55 +1,5 @@ | ||
| 1 | - sitemap: https://moz.com/sitemaps-1-sitemap.xml | |
| 2 | - Sitemap: https://moz.com/community/q/sitemap.xml | |
| 3 | - | |
| 4 | - | |
| 5 | - User-agent: rogerbot | |
| 6 | - Crawl-delay: 10 | |
| 7 | - | |
| 8 | - User-agent: AhrefsBot | |
| 9 | - Crawl-delay: 10 | |
| 10 | - | |
| 11 | - User-agent: GPTBot | |
| 12 | - Disallow: /blog/ | |
| 13 | - Disallow: /learn/seo/ | |
| 14 | - | |
| 15 | - User-agent: DDM-DCipher | |
| 16 | - Disallow: /blog/ | |
| 17 | - Disallow: /learn/seo/ | |
| 18 | - | |
| 1 | + # You are curently using a servd.dev domain to access this file | |
| 2 | + # In order to protect your project's SEO standing, we block spiders from crawling any pages on this domain | |
| 3 | + # You can read more about this here: https://servd.host/docs/unexpected-robots-txt-content | |
| 19 | 4 | User-agent: * |
| 20 | - Allow: /researchtools/ose/$ | |
| 21 | - Allow: /researchtools/ose/dotbot$ | |
| 22 | - Allow: /researchtools/ose/links$ | |
| 23 | - Allow: /researchtools/ose/just-discovered$ | |
| 24 | - Allow: /researchtools/ose/pages$ | |
| 25 | - Allow: /researchtools/ose/domains$ | |
| 26 | - Allow: /researchtools/ose/anchors$ | |
| 27 | - Allow: /products/ | |
| 28 | - Allow: /local/ | |
| 29 | - Allow: /learn/ | |
| 30 | - Allow: /researchtools/ose/ | |
| 31 | - Allow: /researchtools/ose/dotbot$ | |
| 32 | - | |
| 33 | - Disallow: /products/content/ | |
| 34 | - Disallow: /local/enterprise/confirm | |
| 35 | - Disallow: /researchtools/ose/ | |
| 36 | - Disallow: /page-strength/* | |
| 37 | - Disallow: /thumbs/* | |
| 38 | - Disallow: /api/user?* | |
| 39 | - Disallow: /checkout/* | |
| 40 | - Disallow: /purchase/* | |
| 41 | - Disallow: /local/search/ | |
| 42 | - Disallow: /local/details/ | |
| 43 | - Disallow: /messages/ | |
| 44 | - Disallow: /content/audit/* | |
| 45 | - Disallow: /content/search/* | |
| 46 | - Disallow: /marketplace/ | |
| 47 | - Disallow: /cpresources/ | |
| 48 | - Disallow: /vendor/ | |
| 49 | - Disallow: /community/q/questions/*/view_counts | |
| 50 | - Disallow: /admin-preview/* | |
| 51 | - Disallow: /community/q/socket.io/* | |
| 52 | - | |
| 53 | - # File last updated May 5, 2025 | |
| 54 | - | |
| 55 | - | |
| 5 | + Disallow: / |