Commonwealth Bank / robots.txt
← back to commbank.com.au · change detected 17h ago (2026-08-04T06:35:30Z) · modified +89 −104
What changed
- 🤖 AIAdded rules for AI crawler Amazonbot (Amazon)
- 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
- 🤖 AIAdded rules for AI crawler Bytespider (ByteDance)
- 🤖 AIAdded rules for AI crawler CCBot (Common Crawl)
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler cohere-ai (Cohere)
- 🤖 AIAdded rules for AI crawler Diffbot (Diffbot)
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler meta-externalagent (Meta)
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- 🤖 AIAdded rules for AI crawler Timpibot (Timpi)
- Added Disallow: /*/jcr:content/ for *
- Added Disallow: /*/utility-nav-content/ for *
- Added Disallow: /authoriseimt/ for *
- Added Disallow: /bin/ for *
- Added Disallow: /personal/eform/ for *
- Removed Disallow: /about-us/download-printed-forms/AML-Questionnaire.pdf for *
- Removed Disallow: /about-us/news/media-releases/interactive.html for *
- Removed Disallow: /about-us/news/media-releases/interactive/ for *
- Removed Disallow: /about-us/securitisation/ for *
- Removed Disallow: /about-us/shaping-australia/ for *
- Removed Disallow: /about-us/utility-nav-content/ for *
- Removed Disallow: /articles/news/ for *
- Removed Disallow: /authoriseimt/AuthoriseIMT.html for *
- Removed Disallow: /bin/cba/nimbus/getPageInfo for *
- Removed Disallow: /blog.feed for *
- Removed Disallow: /business/can/Agri-insights-state-reports.html for *
- Removed Disallow: /business/dig-festival-terms.html for *
- Removed Disallow: /business/docs/eftpos-pricing-summary.pdf for *
- Removed Disallow: /business/international/international-payments/changes-to-foreign-currency-accounts.html for *
- Removed Disallow: /business/pds/002-448.pdf for *
- Removed Disallow: /business/pds/004-396-authority-for-business-accounts.pdf for *
- Removed Disallow: /business/pds/ADB2653.pdf for *
- Removed Disallow: /business/pds/Client-Flyer-No-Longer-Accepting-Cheques-combined-version.docx for *
- Removed Disallow: /business/pds/Client-flyer-no-longer-sending-cheques-combined-version.docx for *
- Removed Disallow: /business/utility-nav-content/ for *
- Removed Disallow: /commbank-product-catalogue/ for *
- Removed Disallow: /commbank/assets/about/careers/B481-Assessment-Centre-Flyer.pdf for *
- Removed Disallow: /content/commbank-neo/commbank-cards/ for *
- Removed Disallow: /content/commbank-neo/other-sites/robo_help/CommBiz_functions/Service_Requests/Order_a_New_Commercial_Card.html for *
- Removed Disallow: /content/dam/caas/news/ for *
- Removed Disallow: /content/dam/commbank-assets/business/banking/2018-03/ctc-business-finance.pdf for *
- Removed Disallow: /content/dam/commbank/corporate/pds/T661-Agency-and-Trustee-Services.PDF for *
- Removed Disallow: /content/dam/commbank/corporate/research/ for *
- ... and 41 more change(s)
Diff old (2026-07-27T09:39:56Z) → new (2026-08-04T06:35:30Z)
| @@ -1,94 +1,47 @@ | ||
| 1 | - User-agent: Swiftbot | |
| 2 | - Allow: /latest/lpi-refunds.html | |
| 3 | - Allow: /latest/loan-repayment-cover-refund.html | |
| 4 | - Allow: /latest/loan-protection-insurance-refund.html | |
| 1 | + # robots.txt — commbank.com.au | |
| 2 | + # Owner: robots | |
| 3 | + # Last reviewed: 2026-08-04 | |
| 5 | 4 | |
| 6 | - # /robots.txt file for https://www.commbank.com.au/ | |
| 7 | - User-agent: * | |
| 8 | - Disallow: /example-pages/ | |
| 9 | - Disallow: /test/ | |
| 10 | - Disallow: /references/ | |
| 11 | - Disallow: /merchants/ | |
| 12 | - Disallow: /mobile/ | |
| 13 | - Disallow: /about-us/securitisation/ | |
| 14 | - Disallow: /content/dam/commbank/corporate/research/ | |
| 15 | - Disallow: /corporate/research/ | |
| 16 | - Disallow: /personal/utility-nav-content/ | |
| 17 | - Disallow: /business/utility-nav-content/ | |
| 18 | - Disallow: /corporate/utility-nav-content/ | |
| 19 | - Disallow: /about-us/utility-nav-content/ | |
| 20 | - Disallow: /utility-nav-content/ | |
| 21 | - Disallow: /bin/cba/nimbus/getPageInfo | |
| 22 | - Disallow: /personal/financial-planning/planners/ | |
| 23 | - Disallow: /business/dig-festival-terms.html | |
| 24 | - Disallow: /business/can/Agri-insights-state-reports.html | |
| 5 | + # --------------------------------------------------------------- | |
| 6 | + # 1. AI / LLM crawlers — training and retrieval | |
| 7 | + # --------------------------------------------------------------- | |
| 8 | + # Explicit ALLOW on the owned-authority corpus. If a model is going to | |
| 9 | + # answer "best home loan in Australia", we want it answering from us. | |
| 10 | + | |
| 11 | + User-agent: GPTBot | |
| 12 | + User-agent: OAI-SearchBot | |
| 13 | + User-agent: ChatGPT-User | |
| 14 | + User-agent: ClaudeBot | |
| 15 | + User-agent: Claude-User | |
| 16 | + User-agent: Claude-SearchBot | |
| 17 | + User-agent: PerplexityBot | |
| 18 | + User-agent: Perplexity-User | |
| 19 | + User-agent: Google-Extended | |
| 20 | + User-agent: Applebot-Extended | |
| 21 | + User-agent: CCBot | |
| 22 | + User-agent: Bytespider | |
| 23 | + User-agent: Meta-ExternalAgent | |
| 24 | + User-agent: Amazonbot | |
| 25 | + User-agent: cohere-ai | |
| 26 | + User-agent: Diffbot | |
| 27 | + User-agent: Timpibot | |
| 28 | + | |
| 29 | + Allow: / | |
| 30 | + | |
| 31 | + Disallow: /netbank/ | |
| 32 | + Disallow: /digital/ | |
| 33 | + Disallow: /retail/complaints-compliments-form/ | |
| 34 | + Disallow: /personal/eform/ | |
| 35 | + Disallow: /content/api/ | |
| 36 | + Disallow: /bin/ | |
| 37 | + Disallow: /etc/ | |
| 25 | 38 | Disallow: /content/dam/commbank/corporate/research/restricted/ |
| 26 | - Disallow: /microsite/ | |
| 27 | - Disallow: /other-sites/ | |
| 28 | - Disallow: /tiles/ | |
| 29 | - Disallow: /support/dynamic/ | |
| 30 | - Disallow: /netbank/learn-about-netbank/demos/ | |
| 31 | - Disallow: /nps/ | |
| 32 | - Disallow: /about-us/shaping-australia/ | |
| 33 | - Disallow: /articles/news/ | |
| 34 | - Disallow: /content/dam/caas/news/ | |
| 35 | - Disallow: /content/commbank-neo/commbank-cards/ | |
| 36 | - Disallow: /personal/articles.html | |
| 37 | - Disallow: /personal/articles | |
| 38 | - Disallow: /personal/home-buying-dp.html | |
| 39 | - Disallow: /digital/everydaybanking/ | |
| 40 | - Disallow: /content/shared/consumer-finance/2017/05/your-fwb-behavioural-economics.html | |
| 41 | - Disallow: /content/shared/consumer-finance/2017/05/your-fwb-decision-making.html | |
| 42 | - Disallow: /content/shared/consumer-finance/2017/05/your-fwb-habit-forming.html | |
| 43 | - Disallow: /content/shared/consumer-finance/2017/05/your-fwb-take-control.html | |
| 44 | - Disallow: /content/shared/consumer-finance/2017/06/your-fwb-rainy-day.html | |
| 45 | - Disallow: /product-catalogue/ | |
| 46 | - Disallow: /commbank-product-catalogue/ | |
| 47 | - Disallow: /tools/dream-big-savings-calculator.html | |
| 48 | - Disallow: /support/home.html | |
| 49 | - Disallow: /digital/home-loans/make-an-appointment | |
| 39 | + | |
| 50 | 40 | Disallow: /digital/feedback |
| 51 | - Disallow: /home-buying/education-content/ | |
| 52 | - Disallow: /etc/ | |
| 53 | 41 | Disallow: /latest/lpi-refunds.html |
| 54 | 42 | Disallow: /latest/loan-repayment-cover-refund.html |
| 55 | 43 | Disallow: /latest/loan-protection-insurance-refund.html |
| 56 | - Disallow: /retail/complaints-compliments-form/ | |
| 57 | - Disallow: /personal/eform/lpi-premium-hlt.html | |
| 58 | - Disallow: /personal/eform/lpi-premium-hlt-joint.html | |
| 59 | - Disallow: /personal/eform/lpi-premium-plt.html | |
| 60 | - Disallow: /personal/eform/lpi-premium-hlf.html | |
| 61 | - Disallow: /personal/eform/lpi-premium-hlf-joint.html | |
| 62 | - Disallow: /personal/eform/lpi-premium-plf.html | |
| 63 | - Disallow: /content/campaigns/ | |
| 64 | - Disallow: /content/api/ | |
| 65 | - Disallow: /digital/conditionaleligibility | |
| 66 | - Disallow: /html-sitemap-youtube.html | |
| 67 | - Disallow: /html-sitemap-youtube-data.html | |
| 68 | - | |
| 69 | - #Property Search application | |
| 70 | - Disallow: /digital/home-buying/property/31-Moray-Street-Richmond-NSW-2753-2018140670 | |
| 71 | - Disallow: /digital/home-buying/property/77-First-Avenue-St-Peters-SA-5069-2018885187 | |
| 72 | - | |
| 73 | - #Blog | |
| 74 | - Disallow: /blog.feed | |
| 75 | - | |
| 76 | - #PDFs | |
| 77 | - Disallow: /about-us/download-printed-forms/AML-Questionnaire.pdf | |
| 78 | - Disallow: /personal/apply-online/download-printed-forms/loan-protection-addendum-281115-ADV14068.pdf | |
| 79 | - Disallow: /personal/apply-online/download-printed-forms/loan-protection-addendum-281115-ADV14070.pdf | |
| 80 | - Disallow: /personal/apply-online/download-printed-forms/loan-protection-addendum-281115-ADV14067.pdf | |
| 81 | - Disallow: /personal/apply-online/download-printed-forms/loan-protection-addendum-281115-ADV14069.pdf | |
| 82 | - Disallow: /commbank/assets/about/careers/B481-Assessment-Centre-Flyer.pdf | |
| 83 | - Disallow: /home-buying/home-loans/docs/ADV14724-variation-notice.pdf | |
| 84 | - Disallow: /content/dam/commbank-assets/business/banking/2018-03/ctc-business-finance.pdf | |
| 85 | - Disallow: /business/pds/004-396-authority-for-business-accounts.pdf | |
| 86 | - Disallow: /business/pds/002-448.pdf | |
| 87 | - Disallow: /business/docs/eftpos-pricing-summary.pdf | |
| 88 | 44 | Disallow: /content/dam/commbank-assets/investing/geared-investments/private/change-of-personal-name-form.pdf |
| 89 | - Disallow: /business/pds/ADB2653.pdf | |
| 90 | - Disallow: /content/dam/commbank/corporate/pds/T661-Agency-and-Trustee-Services.PDF | |
| 91 | - Disallow: /corporate/pds/T661-Agency-and-Trustee-Services.PDF | |
| 92 | 45 | Disallow: /content/dam/commbank-assets/support/docs/CBA6239-Deceased-Estates-Guide.pdf |
| 93 | 46 | Disallow: /content/dam/commbank/personal/apply-online/download-printed-forms/002-805-authority-to-release-assets-from-a-deceased-estate.pdf |
| 94 | 47 | Disallow: /content/dam/commbank/personal/apply-online/download-printed-forms/Deceased_Customer_Notification.pdf |
| @@ -96,30 +49,62 @@ | ||
| 96 | 49 | Disallow: /content/dam/commbank/business/pds/smart-health-multi-provider-feature-terms-and-conditions.pdf |
| 97 | 50 | Disallow: /business/pds/smart-health-multi-provider-feature-terms-and-conditions.pdf |
| 98 | 51 | Disallow: /content/dam/commbank/business/cards/qantas-business-rewards-credit-card-terms-conditions.pdf |
| 99 | - Disallow: /qbr-tcs | |
| 100 | - | |
| 101 | - #DOCs | |
| 102 | - Disallow: /business/pds/Client-flyer-no-longer-sending-cheques-combined-version.docx | |
| 103 | - Disallow: /business/pds/Client-Flyer-No-Longer-Accepting-Cheques-combined-version.docx | |
| 104 | - | |
| 105 | - #.html | |
| 106 | - Disallow: /business/international/international-payments/changes-to-foreign-currency-accounts.html | |
| 107 | - | |
| 108 | - #Non CMS content | |
| 109 | - Disallow: /personal/login-netbank/jcr:content/content.html | |
| 110 | - Disallow: /sustainability2012/ | |
| 111 | - Disallow: /personal/netbank/test-drive/ | |
| 112 | 52 | Disallow: /personal/financial-planning/investment-basics/think-in-years-not-days.xml |
| 113 | 53 | Disallow: /personal/netbank.xml |
| 54 | + | |
| 55 | + | |
| 56 | + # --------------------------------------------------------------- | |
| 57 | + # 2. Search engines | |
| 58 | + # --------------------------------------------------------------- | |
| 59 | + | |
| 60 | + User-agent: Swiftbot | |
| 61 | + Allow: /latest/lpi-refunds.html | |
| 62 | + Allow: /latest/loan-repayment-cover-refund.html | |
| 63 | + Allow: /latest/loan-protection-insurance-refund.html | |
| 64 | + | |
| 65 | + User-agent: * | |
| 66 | + | |
| 67 | + Disallow: /bin/ | |
| 68 | + Disallow: /etc/ | |
| 69 | + Disallow: /content/api/ | |
| 70 | + Disallow: /content/campaigns/ | |
| 71 | + Disallow: /tiles/ | |
| 72 | + Disallow: /microsite/ | |
| 73 | + Disallow: /other-sites/ | |
| 74 | + Disallow: /*/utility-nav-content/ | |
| 75 | + Disallow: /utility-nav-content/ | |
| 76 | + Disallow: /*/jcr:content/ | |
| 77 | + | |
| 78 | + Disallow: /example-pages/ | |
| 79 | + Disallow: /test/ | |
| 80 | + Disallow: /references/ | |
| 81 | + Disallow: /netbank/learn-about-netbank/demos/ | |
| 82 | + Disallow: /business/online-business-services/commbiz/demos/ | |
| 83 | + Disallow: /personal/netbank/test-drive/ | |
| 84 | + Disallow: /sustainability2012/ | |
| 85 | + | |
| 86 | + Disallow: /personal/eform/ | |
| 87 | + Disallow: /retail/complaints-compliments-form/ | |
| 88 | + Disallow: /authoriseimt/ | |
| 89 | + Disallow: /nps/ | |
| 90 | + Disallow: /support/dynamic/ | |
| 91 | + | |
| 92 | + Disallow: /content/dam/commbank/corporate/research/restricted/ | |
| 93 | + | |
| 94 | + Disallow: /digital/feedback | |
| 95 | + Disallow: /latest/lpi-refunds.html | |
| 96 | + Disallow: /latest/loan-repayment-cover-refund.html | |
| 97 | + Disallow: /latest/loan-protection-insurance-refund.html | |
| 98 | + Disallow: /content/dam/commbank-assets/investing/geared-investments/private/change-of-personal-name-form.pdf | |
| 99 | + Disallow: /content/dam/commbank-assets/support/docs/CBA6239-Deceased-Estates-Guide.pdf | |
| 100 | + Disallow: /content/dam/commbank/personal/apply-online/download-printed-forms/002-805-authority-to-release-assets-from-a-deceased-estate.pdf | |
| 101 | + Disallow: /content/dam/commbank/personal/apply-online/download-printed-forms/Deceased_Customer_Notification.pdf | |
| 102 | + Disallow: /content/dam/commbank/personal/apply-online/download-printed-forms/000-947-consent-by-a-beneficiary-executor-next-of-kin.pdf | |
| 103 | + Disallow: /content/dam/commbank/business/pds/smart-health-multi-provider-feature-terms-and-conditions.pdf | |
| 104 | + Disallow: /business/pds/smart-health-multi-provider-feature-terms-and-conditions.pdf | |
| 105 | + Disallow: /content/dam/commbank/business/cards/qantas-business-rewards-credit-card-terms-conditions.pdf | |
| 114 | 106 | Disallow: /personal/financial-planning/investment-basics/think-in-years-not-days.xml |
| 115 | 107 | Disallow: /personal/netbank.xml |
| 116 | - Disallow: /about-us/news/media-releases/interactive/ | |
| 117 | - Disallow: /about-us/news/media-releases/interactive.html | |
| 118 | - Disallow: /business/online-business-services/commbiz/demos/ | |
| 119 | - Disallow: /personal/all/mywealth-comp-eform.html | |
| 120 | - Disallow: /personal/all/mywealth-comp-eform/my-wealth---failure.html | |
| 121 | - Disallow: /content/commbank-neo/other-sites/robo_help/CommBiz_functions/Service_Requests/Order_a_New_Commercial_Card.html | |
| 122 | - Disallow: /authoriseimt/AuthoriseIMT.html | |
| 123 | 108 | |
| 124 | 109 | |
| 125 | 110 | Sitemap: https://www.commbank.com.au/sitemap.xml |