NeuralCrawl

Ahrefs / robots.txt snapshot

← back to ahrefs.com · fetched 2026-08-07T19:48:33Z (3mo ago) · HTTP 200 · 1038 bytes · sha256 1fe9430b59f5156e · raw

final URL: https://ahrefs.com/robots.txt

1User-agent: *
2Disallow: /article
3Disallow: /site-explorer/ajax/
4Allow: /site-explorer/$
5Disallow: /site-explorer/*
6Allow: /link-intersect/$
7Disallow: /link-intersect/*
8Disallow: /v4*
9Disallow: /blog/*?s=*
10Disallow: /blog/*?archive*
11Disallow: /*/blog
12Allow: /de/blog
13Allow: /es/blog
14Allow: /fr/blog
15Allow: /it/blog
16Allow: /ja/blog
17Allow: /pt/blog
18Allow: /zh/blog
19Disallow: /seo/for/*?*draft
20Disallow: /academy/*?*draft
21Disallow: /seo-toolbar/welcome
22Disallow: /seo-toolbar/uninstall
23Disallow: /*/seo-toolbar/welcome
24Disallow: /*/seo-toolbar/uninstall
25Disallow: /*?input
26Disallow: /draft/*
27Disallow: /academy/draft/*
28Allow: /agencies/*?services[]=*
29Allow: /agencies/*&services[]=*
30Disallow: /agencies/*?*languages[]=*
31Disallow: /agencies/*&*languages[]=*
32Disallow: /agencies/*?*industries[]=*
33Disallow: /agencies/*&*industries[]=*
34Disallow: /agencies/*?*budget=*
35Disallow: /agencies/*&*budget=*
36Disallow: /agencies/*?*businessSize=*
37Disallow: /agencies/*&*businessSize=*
38Disallow: /cdn-cgi/
39Disallow: /writing-tools/experimental-tools.json