Telstra / robots.txt snapshot
← back to telstra.com.au · fetched 2026-06-20T01:10:31Z (18h ago) · HTTP 200 · 3049 bytes · sha256 f0c738e62595167a · raw
final URL: https://www.telstra.com.au/robots.txt
| 1 | Sitemap: http://www.telstra.com.au/sitemap.xml |
| 2 | |
| 3 | |
| 4 | # Block AI bots from all access |
| 5 | |
| 6 | User-agent: Alexa |
| 7 | User-agent: Alexa |
| 8 | User-agent: Amazon-Kendra |
| 9 | User-agent: AmazonBot |
| 10 | User-agent: ByteDance |
| 11 | User-agent: ByteSpider |
| 12 | User-agent: CC-Crawler |
| 13 | User-agent: CCBot |
| 14 | User-agent: ChatGLM |
| 15 | User-agent: ClearScopeBot |
| 16 | User-agent: CommonCrawl |
| 17 | User-agent: DataForSEOBot |
| 18 | User-agent: Diffbot |
| 19 | User-agent: Falcon |
| 20 | User-agent: FlyRiver |
| 21 | User-agent: Goose |
| 22 | User-agent: Grammarly |
| 23 | User-agent: Iaskspider |
| 24 | User-agent: imagesiftBot |
| 25 | User-agent: Img2dataset |
| 26 | User-agent: PetalBot |
| 27 | User-agent: Quillbot |
| 28 | User-agent: Scrapy |
| 29 | User-agent: Sidetrade |
| 30 | User-agent: Sonic |
| 31 | User-agent: TikTokSpider |
| 32 | User-agent: Timpibot |
| 33 | User-agent: VelenPublicWebCrawler |
| 34 | Disallow: / |
| 35 | |
| 36 | |
| 37 | User-agent: Swiftbot |
| 38 | Allow: /aboutus/media/jump-page |
| 39 | |
| 40 | User-agent: Elastic-Crawler Elastic Cloud |
| 41 | Allow: /aboutus/media/jump-page |
| 42 | |
| 43 | User-agent: Elastic-Crawler |
| 44 | Allow: /aboutus/media/jump-page |
| 45 | |
| 46 | User-agent: Elastic Cloud |
| 47 | Allow: /aboutus/media/jump-page |
| 48 | |
| 49 | |
| 50 | User-agent: * |
| 51 | Disallow: /forms |
| 52 | Disallow: /mobile/mobiles/ |
| 53 | Disallow: /uberprod |
| 54 | Disallow: /uberprod/ |
| 55 | Disallow: /idc |
| 56 | Disallow: /idc/ |
| 57 | Disallow: /abouttelstra/media-centre/tv-reel |
| 58 | Disallow: /abouttelstra/media-centre/tv-reel/ |
| 59 | Disallow: /misc/mobile-site/ |
| 60 | Disallow: /abouttelstra/education-centre/ |
| 61 | Disallow: /styleguide |
| 62 | Disallow: /telstraprepaidplus/myprepaid/ |
| 63 | Disallow: /mobile-phones/product-catalogue/ |
| 64 | Disallow: /small-business/product-catalogue/ |
| 65 | Allow: /misc/mobile-site/index.html |
| 66 | Disallow: /aboutus/media/jump-page |
| 67 | Disallow: /products/ |
| 68 | Disallow: /search |
| 69 | Disallow: /content/dam/tcom/personal/my-account/pdf/chown-teg-form.pdf |
| 70 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_mobility_services.pdf |
| 71 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_networks_and_internet.pdf |
| 72 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_store.pdf |
| 73 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_tracking_and_orders.pdf |
| 74 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_user_management.pdf |
| 75 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_incidents.pdf |
| 76 | Disallow: /content/dam/tcom/business-enterprise/telstra-connect-guides/telstra_connect_guide_billing.pdf |
| 77 | Disallow: *?bvstate |
| 78 | Disallow: /content/dam/tcom/personal/help/pdf/cis-business/mobile/business-critical-information-summary-business-fleet-plus-10.pdf |
| 79 | Disallow: /content/dam/tcom/personal/help/pdf/cis-business/mobile/business-critical-information-summary-business-fleet-plus-40.pdf |
| 80 | Disallow: /content/dam/tcom/help/critical-information-summaries/business/mobile/business-mobile-plans/Telstra-Business-Mobile-Plans.pdf |
| 81 | Disallow: /content/dam/tcom/help/critical-information-summaries/business/mobile/business-mobile-plans/Business-Mobile-Plans.pdf |
| 82 | Disallow: /content/dam/shared-component-assets/tecom/news-and-research/gated-assets/ |
| 83 | Disallow: /content/tcom/tplus/ |
| 84 | Disallow: /tplus |
| 85 |