NeuralCrawl

Oncrawl / robots.txt

← back to oncrawl.com · change detected 22d ago (2026-08-27T14:17:26Z) · modified +2 −29

What changed

Diff old (2026-06-20T14:56:27Z) → new (2026-08-27T14:17:26Z)

@@ -1,32 +1,5 @@
1- User-agent: Googlebot
2- Disallow: /ads.txt
3- Disallow: /wpcb/wp-admin/
4- Disallow: /search/
5- Disallow: /*&sa=*
6- Disallow: /&sa=*
7- Disallow: /?s=*
8- Disallow: /.well-known*
9- Disallow: /wpcb/wp-json/contact-form*
10- Disallow: /wpcb/wp-includes/wlwmanifest.xml
11- Disallow: /web/app/modules/*
12- Disallow: /?author=*
13- Disallow: /*feed/
14- Disallow: /*hubs*
15- Allow: /tag/content-hubs/
16-
171  User-agent: *
18- Disallow: /ads.txt
19- Disallow: /wpcb/wp-admin/
20- Disallow: /search/
21- Disallow: /*&sa=*
22- Disallow: /&sa=*
23- Disallow: /?s=*
24- Disallow: /.well-known*
25- Disallow: /wpcb/wp-json/contact-form*
26- Disallow: /wpcb/wp-includes/wlwmanifest.xml
27- Disallow: /web/app/modules/*
28- Disallow: /?author=*
29-
2+ Disallow: /wp/wp-admin/
3+ Allow: /wp/wp-admin/admin-ajax.php
304 
315  Sitemap: https://www.oncrawl.com/sitemap_index.xml
32- Sitemap: https://fr.oncrawl.com/sitemap_index.xml