NeuralCrawl

Moz / robots.txt

← back to moz.com · change detected 11d ago (2026-09-09T00:14:16Z) · modified +51 −4

What changed

Diff old (2026-09-08T18:06:18Z) → new (2026-09-09T00:14:16Z)

@@ -1,5 +1,52 @@
1- # You are curently using a servd.dev domain to access this file
2- # In order to protect your project's SEO standing, we block spiders from crawling any pages on this domain
3- # You can read more about this here: https://servd.host/docs/unexpected-robots-txt-content
1+
2+ Sitemap: https://moz.com/sitemaps-1-sitemap.xml
3+ Sitemap: https://moz.com/community/q/sitemap.xml
4+
5+ User-agent: rogerbot
6+ Crawl-delay: 10
7+
8+ User-agent: AhrefsBot
9+ Crawl-delay: 10
10+
11+ User-agent: GPTBot
12+ Disallow: /blog/
13+ Disallow: /learn/seo/
14+
15+ User-agent: DDM-DCipher
16+ Disallow: /blog/
17+ Disallow: /learn/seo/
18+
419  User-agent: *
5- Disallow: /
20+ Allow: /researchtools/ose/$
21+ Allow: /researchtools/ose/dotbot$
22+ Allow: /researchtools/ose/links$
23+ Allow: /researchtools/ose/just-discovered$
24+ Allow: /researchtools/ose/pages$
25+ Allow: /researchtools/ose/domains$
26+ Allow: /researchtools/ose/anchors$
27+ Allow: /products/
28+ Allow: /local/
29+ Allow: /learn/
30+ Allow: /researchtools/ose/
31+ Allow: /researchtools/ose/dotbot$
32+ Disallow: /products/content/
33+ Disallow: /local/enterprise/confirm
34+ Disallow: /researchtools/ose/
35+ Disallow: /page-strength/*
36+ Disallow: /thumbs/*
37+ Disallow: /api/user?*
38+ Disallow: /checkout/*
39+ Disallow: /purchase/*
40+ Disallow: /local/search/
41+ Disallow: /local/details/
42+ Disallow: /messages/
43+ Disallow: /content/audit/*
44+ Disallow: /content/search/*
45+ Disallow: /marketplace/
46+ Disallow: /cpresources/
47+ Disallow: /vendor/
48+ Disallow: /community/q/questions/*/view_counts
49+ Disallow: /admin-preview/*
50+ Disallow: /community/q/socket.io/*
51+
52+ # File last updated May 5, 2025