NeuralCrawl

Moz / robots.txt snapshot

← back to moz.com · fetched 2026-09-09T00:14:16Z (11d ago) · HTTP 200 · 1207 bytes · sha256 79bb8772fcda6f8a · raw

final URL: https://moz.com/robots.txt

1
2Sitemap: https://moz.com/sitemaps-1-sitemap.xml
3Sitemap: https://moz.com/community/q/sitemap.xml
4
5User-agent: rogerbot
6Crawl-delay: 10
7
8User-agent: AhrefsBot
9Crawl-delay: 10
10
11User-agent: GPTBot
12Disallow: /blog/
13Disallow: /learn/seo/
14
15User-agent: DDM-DCipher
16Disallow: /blog/
17Disallow: /learn/seo/
18
19User-agent: *
20Allow: /researchtools/ose/$
21Allow: /researchtools/ose/dotbot$
22Allow: /researchtools/ose/links$
23Allow: /researchtools/ose/just-discovered$
24Allow: /researchtools/ose/pages$
25Allow: /researchtools/ose/domains$
26Allow: /researchtools/ose/anchors$
27Allow: /products/
28Allow: /local/
29Allow: /learn/
30Allow: /researchtools/ose/
31Allow: /researchtools/ose/dotbot$
32Disallow: /products/content/
33Disallow: /local/enterprise/confirm
34Disallow: /researchtools/ose/
35Disallow: /page-strength/*
36Disallow: /thumbs/*
37Disallow: /api/user?*
38Disallow: /checkout/*
39Disallow: /purchase/*
40Disallow: /local/search/
41Disallow: /local/details/
42Disallow: /messages/
43Disallow: /content/audit/*
44Disallow: /content/search/*
45Disallow: /marketplace/
46Disallow: /cpresources/
47Disallow: /vendor/
48Disallow: /community/q/questions/*/view_counts
49Disallow: /admin-preview/*
50Disallow: /community/q/socket.io/*
51
52# File last updated May 5, 2025