NeuralCrawl

๐Ÿ‡ฆ๐Ÿ‡บ Westpac

westpac.com.au · National indices · rank #51 · Banking · live robots.txt ↗

AI crawler access (latest snapshot, 2d ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 1670 bytes · sha256 1d613ed6a86e · raw

# robots.txt generated for www.westpac.com.au
User-agent: *
Sitemap: https://www.westpac.com.au/sitemap.xml
Sitemap: https://www.westpac.com.au/news/sitemap.xml
Sitemap: https://www.westpac.com.au/content/dam/public/wbc/seo/sitemap-locator.xml
Disallow: /search/
Disallow: /forms/
Disallow: /Forms/
Disallow: /branch-locator/
Disallow: /personal-banking/westpac-online-sign-in-out/signin/
Disallow: /business-banking/business-banking-online-sign-in-out/signin/
Disallow: /corporate-banking/corporate-online/signin/
Disallow: /business-banking/wintrade/signin/
Disallow: /esi2/wintrade-nz-signin/
Disallow: /esi2/ubs-signin/
Disallow: /esi2/col-nz-signin/
Disallow: /internet/publish.nsf/
Disallow: /page-elements-westpac/promotional-images-pages/
Disallow: /page-elements-westpac/carousel/
Disallow: /page-elements-westpac/call-to-action/
Disallow: /personal-banking/home-loans/special-purpose-home-loans/low-doc-loans/rates/
Disallow: /business-banking/merchant-services/mobile-payments/mobilepayway/
Disallow: /_services/
Disallow: /content/public/wbc/en/_services/serialiser/
Disallow: /_services/serialiser/
Disallow: /content/dam/public/wbc/introducernet/
Disallow: /content/public/wbc/en/deeplink-web-redirect.html
Disallow: /business-banking/westpac-live-video/
Disallow: /content/public/wbc/en/business-banking/westpac-live-video/
Disallow: /locateus-ios*
Disallow: /locateus-android*
Disallow: /disclaimer/
Disallow: /footer-cta/
Disallow: /test-aem-1817/*
Disallow: /wbc-rebrand-hub/*
Disallow: /test/*
Disallow: /*.doc
Disallow: /*.docx
Allow: /content/public/wbc/en/_services/serialiser/calculator/
Allow: /content/public/wbc/en/_services/calcs-forms-tools/

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived