NeuralCrawl

Ahrefs / robots.txt snapshot

← back to ahrefs.com · fetched 2026-11-04T01:20:07Z (3d ago) · HTTP 200 · 862 bytes · sha256 e0432e9363d7c9f8 · raw

final URL: https://ahrefs.com/robots.txt

1User-agent: *
2Disallow: /article
3Disallow: /site-explorer/ajax/
4Allow: /site-explorer/$
5Disallow: /site-explorer/*
6Allow: /link-intersect/$
7Disallow: /link-intersect/*
8Disallow: /v4*
9Disallow: /blog/*?s=*
10Disallow: /blog/*?archive*
11Disallow: /seo/for/*?*draft
12Disallow: /academy/*?*draft
13Disallow: /seo-toolbar/welcome
14Disallow: /seo-toolbar/uninstall
15Disallow: /*/seo-toolbar/welcome
16Disallow: /*/seo-toolbar/uninstall
17Disallow: /*?input
18Allow: /agencies/*?services[]=*
19Allow: /agencies/*&services[]=*
20Disallow: /agencies/*?*languages[]=*
21Disallow: /agencies/*&*languages[]=*
22Disallow: /agencies/*?*industries[]=*
23Disallow: /agencies/*&*industries[]=*
24Disallow: /agencies/*?*budget=*
25Disallow: /agencies/*&*budget=*
26Disallow: /agencies/*?*businessSize=*
27Disallow: /agencies/*&*businessSize=*
28Disallow: /cdn-cgi/
29Disallow: /writing-tools/experimental-tools.json