NeuralCrawl

Moz / robots.txt snapshot

← back to moz.com · fetched 2026-06-20T14:56:27Z (7h ago) · HTTP 200 · 1210 bytes · sha256 07b090a959e9e282 · raw

final URL: https://moz.com/robots.txt

1sitemap: https://moz.com/sitemaps-1-sitemap.xml
2Sitemap: https://moz.com/community/q/sitemap.xml
3
4
5User-agent: rogerbot
6Crawl-delay: 10
7
8User-agent: AhrefsBot
9Crawl-delay: 10
10
11User-agent: GPTBot
12Disallow: /blog/
13Disallow: /learn/seo/
14
15User-agent: DDM-DCipher
16Disallow: /blog/
17Disallow: /learn/seo/
18
19User-agent: *
20Allow: /researchtools/ose/$
21Allow: /researchtools/ose/dotbot$
22Allow: /researchtools/ose/links$
23Allow: /researchtools/ose/just-discovered$
24Allow: /researchtools/ose/pages$
25Allow: /researchtools/ose/domains$
26Allow: /researchtools/ose/anchors$
27Allow: /products/
28Allow: /local/
29Allow: /learn/
30Allow: /researchtools/ose/
31Allow: /researchtools/ose/dotbot$
32
33Disallow: /products/content/
34Disallow: /local/enterprise/confirm
35Disallow: /researchtools/ose/
36Disallow: /page-strength/*
37Disallow: /thumbs/*
38Disallow: /api/user?*
39Disallow: /checkout/*
40Disallow: /purchase/*
41Disallow: /local/search/
42Disallow: /local/details/
43Disallow: /messages/
44Disallow: /content/audit/*
45Disallow: /content/search/*
46Disallow: /marketplace/
47Disallow: /cpresources/
48Disallow: /vendor/
49Disallow: /community/q/questions/*/view_counts
50Disallow: /admin-preview/*
51Disallow: /community/q/socket.io/*
52
53# File last updated May 5, 2025
54
55