NeuralCrawl

๐Ÿ‡ฎ๐Ÿ‡ณ Cricbuzz

cricbuzz.com · Top 1000 websites · rank #105 · Web · live robots.txt ↗

AI crawler access (latest snapshot, 1h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 9658 bytes · sha256 fbd98b09466f · raw

# Cricbuzz - The Interactive Cricket Portal
# Nothing very exciting here for you I'm afraid.
# Despictable and evil robots to keep out :) 
User-agent: *
Allow: /ads.txt
Disallow: /
User-agent: Googlebot
Allow: /ads.txt
Allow: /app-ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Googlebot-Image
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Google-Mobile
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Mediapartners-Google
Allow: /ads.txt
Allow: /app-ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Yahoo
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Slurp
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Bingbot
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: MSNBot
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: MSNBot-Media
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: AdIdxBot
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: BingPreview
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Yandex
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Baidu
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Twitterbot
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Yeti
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Excite
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Duckduckgo
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Lycos
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: Applebot
Allow: /ads.txt
Disallow: /api/
Disallow: /register/
Disallow: /match/
Disallow: /match2/
Disallow: /match3/
Disallow: /match4/
Disallow: /cgi-bin/
Disallow: /protected/
Disallow: /livescore/
Disallow: /component/
Disallow: /livecricketscore/
Disallow: /syndication/
Disallow: /includes/
Disallow: /mobile/subscribe
Disallow: /*.json
Disallow: /*.txt
Disallow: /*.html
Disallow: /*.php
Disallow: /archive/archive/
Disallow: /search?*
Disallow: /search/?*
User-agent: AmazonAdBot
Allow: /ads.txt
Allow: /app-ads.txt
Allow: /
Disallow: /api/
Disallow: /search?*
Disallow: /search/?*
Disallow: /*.json
Sitemap: https://www.cricbuzz.com/sitemap/cricket-videos.xml
Sitemap: https://www.cricbuzz.com/sitemap/live-matches.xml
Sitemap: https://www.cricbuzz.com/sitemap/cricket-schedule-series.xml
Sitemap: https://www.cricbuzz.com/sitemap.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived