NeuralCrawl

Oncrawl / robots.txt snapshot

← back to oncrawl.com · fetched 2026-06-20T14:56:27Z (4h ago) · HTTP 200 · 744 bytes · sha256 2ee140d5bb36a02e · raw

final URL: https://oncrawl.com/robots.txt

1User-agent: Googlebot
2Disallow: /ads.txt
3Disallow: /wpcb/wp-admin/
4Disallow: /search/
5Disallow: /*&sa=*
6Disallow: /&sa=*
7Disallow: /?s=*
8Disallow: /.well-known*
9Disallow: /wpcb/wp-json/contact-form*
10Disallow: /wpcb/wp-includes/wlwmanifest.xml
11Disallow: /web/app/modules/*
12Disallow: /?author=*
13Disallow: /*feed/
14Disallow: /*hubs*
15Allow: /tag/content-hubs/
16
17User-agent: *
18Disallow: /ads.txt
19Disallow: /wpcb/wp-admin/
20Disallow: /search/
21Disallow: /*&sa=*
22Disallow: /&sa=*
23Disallow: /?s=*
24Disallow: /.well-known*
25Disallow: /wpcb/wp-json/contact-form*
26Disallow: /wpcb/wp-includes/wlwmanifest.xml
27Disallow: /web/app/modules/*
28Disallow: /?author=*
29
30
31Sitemap: https://www.oncrawl.com/sitemap_index.xml
32Sitemap: https://fr.oncrawl.com/sitemap_index.xml