Oncrawl / robots.txt snapshot
← back to oncrawl.com · fetched 2026-08-27T14:17:26Z (22d ago) · HTTP 200 · 125 bytes · sha256 7a10a8814cfecb0f · raw
final URL: https://www.oncrawl.com/robots.txt
| 1 | User-agent: * |
| 2 | Disallow: /wp/wp-admin/ |
| 3 | Allow: /wp/wp-admin/admin-ajax.php |
| 4 | |
| 5 | Sitemap: https://www.oncrawl.com/sitemap_index.xml |