NeuralCrawl

Oncrawl / robots.txt

← back to oncrawl.com · change detected 19d ago (2026-08-30T15:17:49Z) · modified +29 −2

What changed

Diff old (2026-08-27T14:17:26Z) → new (2026-08-30T15:17:49Z)

@@ -1,5 +1,32 @@
1+ User-agent: Googlebot
2+ Disallow: /ads.txt
3+ Disallow: /wpcb/wp-admin/
4+ Disallow: /search/
5+ Disallow: /*&sa=*
6+ Disallow: /&sa=*
7+ Disallow: /?s=*
8+ Disallow: /.well-known*
9+ Disallow: /wpcb/wp-json/contact-form*
10+ Disallow: /wpcb/wp-includes/wlwmanifest.xml
11+ Disallow: /web/app/modules/*
12+ Disallow: /?author=*
13+ Disallow: /*feed/
14+ Disallow: /*hubs*
15+ Allow: /tag/content-hubs/
16+
117  User-agent: *
2- Disallow: /wp/wp-admin/
3- Allow: /wp/wp-admin/admin-ajax.php
18+ Disallow: /ads.txt
19+ Disallow: /wpcb/wp-admin/
20+ Disallow: /search/
21+ Disallow: /*&sa=*
22+ Disallow: /&sa=*
23+ Disallow: /?s=*
24+ Disallow: /.well-known*
25+ Disallow: /wpcb/wp-json/contact-form*
26+ Disallow: /wpcb/wp-includes/wlwmanifest.xml
27+ Disallow: /web/app/modules/*
28+ Disallow: /?author=*
29+
430 
531  Sitemap: https://www.oncrawl.com/sitemap_index.xml
32+ Sitemap: https://fr.oncrawl.com/sitemap_index.xml