NeuralCrawl

Barracuda / robots.txt snapshot

← back to barracuda.com · fetched 2026-09-22T04:22:28Z (1mo ago) · HTTP 200 · 2563 bytes · sha256 2bbd92617d6670ae · raw

final URL: https://www.barracuda.com/robots.txt

1User-agent: *
2
3# Prevent Crawling for Thank You Pages
4Disallow: */*LP-OnDemand*
5Disallow: */*LP-ThankYou*
6Disallow: */*thankyou*
7Disallow: */*thank-you*
8Disallow: */thankyou/*
9Disallow: */thankyou*
10Disallow: */thankYou*
11
12# Prevent Crawling for Login & Register
13Disallow: */login/*
14Disallow: */register/*
15Disallow: */customers/s/login/*
16Disallow: */auth/login/*
17
18# Prevent Error Crawling for Templates
19Disallow: /error/
20
21# Prevent Crawling for Captcha
22Disallow: */captcha/*
23
24# Prevent Crawling for Admin
25Disallow: */admin/*
26Disallow: */administrator/*
27
28# Prevent Crawling for User
29Disallow: */user/*
30Disallow: */users/*
31
32# Prevent Crawling for CGI
33Disallow: /cgi-bin/
34
35# Prevent Crawling for cache and log
36Disallow: */cache/*
37Disallow: */log/*
38
39########## SITEMAP FILES ##########
40Sitemap: https://www.barracuda.com/sitemap.xml
41
42########## PREVENT BAD BOTS TO CRAWL ##########
43
44# http://webmeup-crawler.com/
45User-agent: BLEXBot
46Disallow: /
47
48# https://cliqz.com/company/cliqz-bot
49User-agent: Cliqzbot
50Disallow: /
51
52# https://domainstats.com/pages/our-bot
53User-agent: DomainStatsBot
54Disallow: /
55
56# http://www.opensiteexplorer.org/dotbot
57User-agent: DotBot
58User-agent: dotbot
59Disallow: /
60
61# http://www.exalead.com
62User-agent: Exabot
63Disallow: /
64
65# https://developers.ezooms.com/ezooms/ezooms-bot
66User-agent: Ezooms
67Disallow: /
68
69# https://www.gigablast.com/spider.html
70User-agent: Gigabot
71Disallow: /
72
73# https://www.httrack.com/
74User-agent: HTTrack
75Disallow: /
76
77# https://megaindex.com/crawler
78User-agent: MegaIndex.ru
79Disallow: /
80
81User-agent: megaindex.com
82Disallow: /
83
84# https://scoutjet.com/
85User-agent: ScoutJet
86Disallow: /
87
88# http://www.searchmetrics.com
89User-agent: SearchmetricsBot
90Disallow: /
91
92# Disallow: SEOkicks-Robot
93User-agent: SEOkicks
94Disallow:/
95
96User-agent: SEOkicks-Robot
97Disallow:/
98
99# https://napoveda.seznam.cz/en/fulltext-search/crawling/
100User-agent: SeznamBot
101Disallow: /
102
103# Disallow: Sistrix
104User-agent: sistrix
105User-agent: SISTRIX Crawler
106User-agent: SISTRIX
107Disallow: /
108
109# http://help.sogou.com/homepage/search-engine/add-crawl.html
110User-agent: Sogou Spider
111Disallow: /
112
113# http://www.soso.com/help/webmasters.htm
114User-agent : Sosospider
115Disallow: /
116
117# http://openlinkprofiler.org/bot
118User-agent: spbot
119Disallow: /
120
121# http://www.domaintools.com/webmasters/surveybot.php
122User-agent: SurveyBot
123Disallow: /
124
125# https://teleport.dev/
126User-agent: Teleport
127Disallow: /
128
129# http://www.tenmax.com/teleportpro/
130User-agent: TeleportPro
131Disallow: /
132
133# https://www.youdao.com/help/webmaster/spider/
134User-agent: YoudaoBot
135Disallow: /
136
137########## END ##########