NeuralCrawl

πŸ‡ΊπŸ‡Έ Charter Communications

charter.com · Top 1000 websites · rank #864 · Telecommunications · live robots.txt ↗

AI crawler access (latest snapshot, 13h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 2711 bytes · sha256 fa96650cf17c · raw

Sitemap: https://www.spectrum.com/sitemap-index.xml
Sitemap: https://www.globalsiteseo.com/spectrum.GSM/spectrum.gsm.index.en_super.xml
Sitemap: https://www.globalsiteseo.com/spectrum.GSM/spectrum.gsm.index.es_super.xml

User-agent: *

Crawl-delay: 10


# Allowed Paths
Allow: /etc/clientlibs/*.css #
Allow: /etc/designs/*.css #
Allow: /etc/clientlibs/*.js #
Allow: /content/*.json #
Allow: /content/dam/*.js #
Allow: /content/dam/*.css #
Allow: /content/*.svg #
Allow: /content/dam/*.png #
Allow: /content/dam/*.jpg #
Allow: /assets/images/*.webp #
Allow: /ads.txt  #
Allow: /app-ads.txt  #
# Additional Allowed Enterprise
Allow: /etc/cloudsettings/default/contexthub.kernel.js #
Allow: /etc/designs/spectrum/enterprise/favicon.ico #
Allow: /etc.clientlibs/*.css #
Allow: /etc.clientlibs/*.js #


# Excluded Pages
Disallow: /content/dam/spectrum/residential/en/pdfs/policies/MutualArbitrationAgreement.pdf.html #
Disallow: /content/dam/spectrum/residential/en/pdfs/policies/SolutionChannelDocument.pdf.html #
Disallow: /spectrum-home-ec.html #
# Additional Excluded Enterprise
Disallow: /search.html #


# Excluded Tags



# Excluded Paths
Disallow: /*cmp=sod #
Disallow: /*cmp=sspp #
Disallow: /*[object #
Disallow: /content/spectrum/util/spectrum/residential/xref/ #
Disallow: /content/spectrum/residential/fed/ #
Disallow: /content/spectrum/residential/qa/ #
Disallow: /browse/content/dynaTraceMonitor? #
Disallow: /content/campaigns/* #
Disallow: /content/campaigns/spectrum-residential/ #
Disallow: /content/spectrum/residential/microsites/choice-tv/ #
Disallow: /content/codered/* #
Disallow: /content/spectrum/business/en/404 #
Disallow: /content/spectrum/business/en/maintenance #
Disallow: /content/spectrum/business/en/global-elements #
Disallow: /content/spectrum/business/en/company #
Disallow: /content/spectrum/business/en/templates #
Disallow: /content/spectrum/business/fed #
Disallow: /content/spectrum/business/qa-testing-sandbox #
Disallow: /content/spectrum/business/microsites #
Disallow: /content/campaigns/spectrum-business #
Disallow: /homepage* #
Disallow: /mobile/login/ #
Disallow: /content/ #
Disallow: /browse/ #
Disallow: /mobile/homepage* #
Disallow: /business/sales #
Disallow: /test #
Disallow: /campaign-lp/* #
Disallow: *spectrumbusiness-home* #
# Excluded Enterprise Paths
Disallow: /search #
Disallow: /content/spectrum/enterprise/en/home/resource-center #
Disallow: /content/spectrum/enterprise/en/home/services #
Disallow: /content/spectrum/enterprise/en/home/playground #
Disallow: /content/spectrum/enterprise/en/home/se-internal #
Disallow: /content/spectrum/enterprise/en/home/about-old #
Disallow: /content/spectrum/enterprise/en/home/exp-frags #
Disallow: /etc  #

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived