NeuralCrawl

🇺🇸 Barracuda

barracuda.com · Cybersecurity · rank #34 · Cybersecurity · live robots.txt ↗

AI crawler access (latest snapshot, 2d ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅Meta-WebIndexer
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot

Current robots.txt 5066 bytes · sha256 72e62ffd6dd1 · raw 

User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: *

# Prevent Crawling for Thank You Pages
Disallow: */*LP-OnDemand*
Disallow: */*LP-ThankYou*
Disallow: */*thankyou*
Disallow: */*thank-you*
Disallow: */thankyou/*
Disallow: */thankyou*
Disallow: */thankYou*

# Prevent Crawling for Login & Register
Disallow: */login/*
Disallow: */register/*
Disallow: */customers/s/login/*
Disallow: */auth/login/*

# Prevent Error Crawling for Templates
Disallow: /error/

# Prevent Crawling for Captcha
Disallow: */captcha/*

# Prevent Crawling for Admin
Disallow: */admin/*
Disallow: */administrator/*

# Prevent Crawling for User
Disallow: */user/*
Disallow: */users/*

# Prevent Crawling for CGI
Disallow: /cgi-bin/

# Prevent Crawling for cache and log
Disallow: */cache/*
Disallow: */log/*


########## SITEMAP FILES ##########
Sitemap: https://www.barracuda.com/sitemap.xml



########## PREVENT BAD BOTS TO CRAWL ##########

# http://www.opensiteexplorer.org/dotbot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: DotBot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: dotbot
Disallow: /

# http://help.sogou.com/homepage/search-engine/add-crawl.html
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Sogou Spider
Disallow: /

# http://www.exalead.com
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Exabot
Disallow: /

# https://www.youdao.com/help/webmaster/spider/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: YoudaoBot
Disallow: /

# https://megaindex.com/crawler
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: MegaIndex.ru
Disallow: /

User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: megaindex.com
Disallow: /

# https://napoveda.seznam.cz/en/fulltext-search/crawling/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SeznamBot
Disallow: /

# https://cliqz.com/company/cliqz-bot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Cliqzbot
Disallow: /

# https://www.gigablast.com/spider.html
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Gigabot
Disallow: /

# http://openlinkprofiler.org/bot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: spbot
Disallow: /

# https://developers.ezooms.com/ezooms/ezooms-bot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Ezooms
Disallow: /

# https://scoutjet.com/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: ScoutJet
Disallow: /

# http://webmeup-crawler.com/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: BLEXBot
Disallow: /

# Disallow: Sistrix
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: sistrix
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SISTRIX Crawler
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SISTRIX
Disallow: /

# Disallow: SEOkicks-Robot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SEOkicks
Disallow:/

User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SEOkicks-Robot
Disallow:/

# http://www.searchmetrics.com
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SearchmetricsBot
Disallow: /

# http://www.domaintools.com/webmasters/surveybot.php
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: SurveyBot
Disallow: /

# http://cliqz.com/company/cliqzbot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Cliqzbot
Disallow: /

# https://domainstats.com/pages/our-bot
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: DomainStatsBot
Disallow: /

# http://www.soso.com/help/webmasters.htm
User-agent: Sosospider
Disallow: /

# https://teleport.dev/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: Teleport
Disallow: /

# http://www.tenmax.com/teleportpro/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: TeleportPro
Disallow: /

# https://www.httrack.com/
User-agent: *
Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html
Crawl-delay: 1
User-agent: HTTrack
Disallow: /

########## END ##########

Compare any two snapshots

Change history

  1. modified +27 −34 view diff → snapshot
    • 🤖 AIUnblocked CCBot (Common Crawl) - rules removed
    • Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for *
    • Removed Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for *
    • Added Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for DotBot
    • Removed Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for DotBot
    • … 4 more
  2. created +28 −28 view diff → snapshot
    • Added Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for *
    • Removed Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for *
    • Added Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for DotBot
    • Removed Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for DotBot
    • Added Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for sistrix
    • … 3 more
  3. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  4. created +28 −28 view diff → snapshot
    • Added Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for *
    • Removed Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for *
    • Added Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for DotBot
    • Removed Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for DotBot
    • Added Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for sistrix
    • … 3 more
  5. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  6. created +28 −28 view diff → snapshot
    • Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for *
    • Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for *
    • Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for DotBot
    • Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for DotBot
    • Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for sistrix
    • … 3 more
  7. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  8. created +0 −0 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  9. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  10. created +28 −28 view diff → snapshot
    • Added Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for *
    • Removed Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for *
    • Added Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for DotBot
    • Removed Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for DotBot
    • Added Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for sistrix
    • … 3 more
  11. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  12. created +28 −28 view diff → snapshot
    • Added Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for *
    • Removed Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for *
    • Added Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for DotBot
    • Removed Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for DotBot
    • Added Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for sistrix
    • … 3 more
  13. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  14. created +28 −28 view diff → snapshot
    • Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for *
    • Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
    • Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for DotBot
    • Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
    • Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for sistrix
    • … 3 more
  15. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  16. created +28 −28 view diff → snapshot
    • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
    • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
    • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
    • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
    • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for sistrix
    • … 3 more
  17. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  18. created +0 −0 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  19. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  20. created +28 −28 view diff → snapshot
    • Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
    • Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for *
    • Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
    • Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for DotBot
    • Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for sistrix
    • … 3 more
  21. removed +0 −232
    • robots.txt is no longer served (HTTP 404)
  22. initial snapshot
    • First snapshot of robots.txt archived