NeuralCrawl

Similarweb / robots.txt snapshot

← back to similarweb.com · fetched 2026-06-20T14:56:27Z (7h ago) · HTTP 200 · 1026 bytes · sha256 f8b547fa4a960bcc · raw

final URL: https://www.similarweb.com/robots.txt

1User-agent: *
2Allow: /corp/search*
3Allow: /corp/*/search*
4Disallow: */search/*
5Disallow: */adult/*
6Disallow: /corp/*.pdf$
7Disallow: /corp/solution/
8Disallow: /corp/lps/
9Disallow: /corp/get-data/
10Disallow: /corp/unlock-growth/
11Disallow: /silent-login/
12Disallow: /signin-oidc/
13Disallow: /signout-oidc/
14#LLMs-txt: https://www.similarweb.com/llms.txt
15Sitemap: https://www.similarweb.com/corp/sitemap_index.xml
16Sitemap: https://www.similarweb.com/blog/sitemap_index.xml
17Sitemap: https://www.similarweb.com/sitemaps/sitemap_index.xml.gz
18#
19# sMMMMMMMMs
20# MNdmMh+-``.:ohNM
21# MNy/ .sMd- `/yNM
22# MNo` sMo .oNM
23# Md - sMo -dM
24# 'MM+ -dMm+. yMM'
25# MN` `:yNMd/ .NM
26# MN- `-hMy :MM
27# MMN- sMd` -NM
28# Md:` `sh`` .sMM
29# Mdms+-.```-hmMds
30# oMMMMMMMMo
31#
32# OFFICIAL MEASURE OF THE DIGITAL WORLD
33#