NeuralCrawl

Toronto Star / robots.txt snapshot

← back to thestar.com · fetched 2026-08-04T18:49:16Z (5h ago) · HTTP 200 · 12006 bytes · sha256 9142dc29d6191f6b · raw

final URL: https://thestar.com/robots.txt

1User-agent: AI2Bot
2Disallow: /
3
4User-agent: Ai2Bot-Dolma
5Disallow: /
6
7User-agent: Applebot-Extended
8Disallow: /
9
10User-agent: Bytespider
11Disallow: /
12
13User-agent: CCBot
14Disallow: /
15
16User-agent: ChatGPT-User
17Disallow: /
18
19User-agent: Claude-Web
20Disallow: /
21
22User-agent: ClaudeBot
23Disallow: /
24
25User-agent: Diffbot
26Disallow: /
27
28User-agent: FacebookBot
29Disallow: /
30
31User-agent: FriendlyCrawler
32Disallow: /
33
34User-agent: GPTBot
35Disallow: /
36
37User-agent: Google-Extended
38Disallow: /
39
40User-agent: ICC-Crawler
41Disallow: /
42
43User-agent: ImagesiftBot
44Disallow: /
45
46User-agent: Meta-ExternalAgent
47Disallow: /
48
49User-agent: OAI-SearchBot
50Disallow: /
51
52User-agent: PerplexityBot
53Disallow: /
54
55User-agent: PetalBot
56Disallow: /
57
58User-agent: Scrapy
59Disallow: /
60
61User-agent: Timpibot
62Disallow: /
63
64User-agent: VelenPublicWebCrawler
65Disallow: /
66
67User-agent: Webzio-Extended
68Disallow: /
69
70User-agent: YouBot
71Disallow: /
72
73User-agent: anthropic-ai
74Disallow: /
75
76User-agent: cohere-ai
77Disallow: /
78
79User-agent: iaskspider/2.0
80Disallow: /
81
82User-agent: img2dataset
83Disallow: /
84
85User-agent: omgili
86Disallow: /
87
88User-agent: omgilibot
89Disallow: /
90
91User-agent: cohere-training-data-crawler
92Disallow: /
93
94User-agent: Kangaroo Bot
95Disallow: /
96
97User-agent: PanguBot
98Disallow: /
99
100User-agent: Amazonbot
101Disallow: /
102
103User-agent: DuckAssistBot
104Disallow: /
105
106User-agent: Meta-ExternalFetcher
107Disallow: /
108
109User-agent: Perplexity-User
110Disallow: /
111
112User-agent: Claude-SearchBot
113Disallow: /
114
115User-agent: Claude-User
116Disallow: /
117
118User-agent: Cotoyogi
119Disallow: /
120
121User-agent: Datenbank
122Disallow: /
123
124User-agent: Devin
125Disallow: /
126
127User-agent: Factset_spyderbot
128Disallow: /
129
130User-agent: Gemini-Deep-Research
131Disallow: /
132
133User-agent: MistralAI-User
134Disallow: /
135
136User-agent: netEstate
137Disallow: /
138
139User-agent: Imprint
140Disallow: /
141
142User-agent: SemrushBot-OCOB
143Disallow: /
144
145User-agent: AddSearchBot
146Disallow: /
147
148User-agent: bigsur.ai
149Disallow: /
150
151User-agent: CloudVertexBot
152Disallow: /
153
154User-agent: GoogleOther
155Disallow: /
156
157User-agent: LinerBot
158Disallow: /
159
160User-agent: meta-externalagent
161Disallow: /
162
163User-agent: QualifiedBot
164Disallow: /
165
166User-agent: Ai2Bot-DeepResearchEval
167Disallow: /
168
169User-agent: ChatGLM-Spider
170Disallow: /
171
172User-agent: DeepSeekBot
173Disallow: /
174
175User-agent: Google-NotebookLM
176Disallow: /
177
178User-agent: iAskBot
179Disallow: /
180
181User-agent: KunatoCrawler
182Disallow: /
183
184User-agent: meta-externalfetcher
185Disallow: /
186
187User-agent: meta-webindexer
188Disallow: /
189
190User-agent: webzio-extended
191Disallow: /
192
193User-agent: WRTNBot
194Disallow: /
195
196User-agent: ZanistaBot
197Disallow: /
198
199User-agent: AI2Bot-DeepResearchEval
200Disallow: /
201
202User-agent: Anomura
203Disallow: /
204
205User-agent: atlassian-bot
206Disallow: /
207
208User-agent: Bravebot
209Disallow: /
210
211User-agent: Channel3Bot
212Disallow: /
213
214User-agent: Cloudflare-AutoRAG
215Disallow: /
216
217User-agent: Crawl4AI
218Disallow: /
219
220User-agent: Google-CloudVertexBot
221Disallow: /
222
223User-agent: imageSpider
224Disallow: /
225
226User-agent: KlaviyoAIBot
227Disallow: /
228
229User-agent: laion-huggingface-processor
230Disallow: /
231
232User-agent: LCC
233Disallow: /
234
235User-agent: LinkupBot
236Disallow: /
237
238User-agent: PhindBot
239Disallow: /
240
241User-agent: Poggio-Citations
242Disallow: /
243
244User-agent: SBIntuitionsBot
245Disallow: /
246
247User-agent: TavilyBot
248Disallow: /
249
250User-agent: Amzn-SearchBot
251Disallow: /
252
253User-agent: Amzn-User
254Disallow: /
255
256User-agent: ApifyBot
257Disallow: /
258
259User-agent: ApifyWebsiteContentCrawler
260Disallow: /
261
262User-agent: Aranet-SearchBot
263Disallow: /
264
265User-agent: AzureAI-SearchBot
266Disallow: /
267
268User-agent: ExaBot
269Disallow: /
270
271User-agent: FirecrawlAgent
272Disallow: /
273
274User-agent: kagi-fetcher
275Disallow: /
276
277User-agent: TerraCotta
278Disallow: /
279
280User-agent: ScraperAPI
281Disallow: /
282
283User-agent: Scraper-API
284Disallow: /
285
286User-agent: ScrapingBee
287Disallow: /
288
289User-agent: ScrapingBeeBot
290Disallow: /
291
292User-agent: ZenRows
293Disallow: /
294
295User-agent: ZenRowsBot
296Disallow: /
297
298User-agent: ScrapeGraphAI
299Disallow: /
300
301User-agent: scrapegraph
302Disallow: /
303
304User-agent: Scrapfly
305Disallow: /
306
307User-agent: ScrapflyBot
308Disallow: /
309
310User-agent: ScrapflyClient
311Disallow: /
312
313User-agent: Zyte
314Disallow: /
315
316User-agent: ZyteBot
317Disallow: /
318
319User-agent: Scrapinghub
320Disallow: /
321
322User-agent: SerpApi
323Disallow: /
324
325User-agent: SerpApiBot
326Disallow: /
327
328User-agent: WebScraping.AI
329Disallow: /
330
331User-agent: WebScrapingAI
332Disallow: /
333
334User-agent: webscraping-ai
335Disallow: /
336
337User-agent: Scrapeless
338Disallow: /
339
340User-agent: ScrapelessBot
341Disallow: /
342
343User-agent: Scrapingdog
344Disallow: /
345
346User-agent: ScrapingdogBot
347Disallow: /
348
349User-agent: Scrapestack
350Disallow: /
351
352User-agent: ScrapestackBot
353Disallow: /
354
355User-agent: ScaleSERP
356Disallow: /
357
358User-agent: SerpWow
359Disallow: /
360
361User-agent: Rainforest-API
362Disallow: /
363
364User-agent: BlueCart-API
365Disallow: /
366
367User-agent: BrightData
368Disallow: /
369
370User-agent: Bright Data
371Disallow: /
372
373User-agent: BrightBot
374Disallow: /
375
376User-agent: Smartproxy
377Disallow: /
378
379User-agent: Oxylabs
380Disallow: /
381
382User-agent: OxylabsBot
383Disallow: /
384
385User-agent: Soax
386Disallow: /
387
388User-agent: SoaxBot
389Disallow: /
390
391User-agent: SOAX
392Disallow: /
393
394User-agent: Nimbleway
395Disallow: /
396
397User-agent: NimbleCrawler
398Disallow: /
399
400User-agent: NimblewayBot
401Disallow: /
402
403User-agent: Froxy
404Disallow: /
405
406User-agent: FroxyBot
407Disallow: /
408
409User-agent: BrowseAI
410Disallow: /
411
412User-agent: Browse-AI
413Disallow: /
414
415User-agent: BrowseAIBot
416Disallow: /
417
418User-agent: Octoparse
419Disallow: /
420
421User-agent: OctoparseBot
422Disallow: /
423
424User-agent: Import.io
425Disallow: /
426
427User-agent: ImportIO
428Disallow: /
429
430User-agent: importio
431Disallow: /
432
433User-agent: Kadoa
434Disallow: /
435
436User-agent: Kadoabot
437Disallow: /
438
439User-agent: KadoaBot
440Disallow: /
441
442User-agent: ScrapeHero
443Disallow: /
444
445User-agent: Scrape-Hero
446Disallow: /
447
448User-agent: ScrapeHeroBot
449Disallow: /
450
451User-agent: Grepsr
452Disallow: /
453
454User-agent: GrepsrBot
455Disallow: /
456
457User-agent: GrepsrCrawler
458Disallow: /
459
460User-agent: MrScraper
461Disallow: /
462
463User-agent: Mr-Scraper
464Disallow: /
465
466User-agent: MrScraperBot
467Disallow: /
468
469User-agent: PromptCloud
470Disallow: /
471
472User-agent: PromptCloudBot
473Disallow: /
474
475User-agent: CrawlNow
476Disallow: /
477
478User-agent: CrawlNowBot
479Disallow: /
480
481User-agent: ForageAI
482Disallow: /
483
484User-agent: Forage-AI
485Disallow: /
486
487User-agent: ForageBot
488Disallow: /
489
490User-agent: DumplingAI
491Disallow: /
492
493User-agent: Dumpling-AI
494Disallow: /
495
496User-agent: DumplingBot
497Disallow: /
498
499User-agent: spider-cloud
500Disallow: /
501
502User-agent: SpiderCloud
503Disallow: /
504
505User-agent: Hyperbrowser
506Disallow: /
507
508User-agent: HyperbrowserBot
509Disallow: /
510
511User-agent: HyperAgent
512Disallow: /
513
514User-agent: skrape.ai
515Disallow: /
516
517User-agent: SkrapeAI
518Disallow: /
519
520User-agent: SkrapeBot
521Disallow: /
522
523User-agent: CloudflareBrowserRenderingCrawler
524Disallow: /
525
526User-agent: Apify-HTTP-Client
527Disallow: /
528
529User-agent: amazon-kendra
530Disallow: /
531
532User-agent: amazon-QBusiness
533Disallow: /
534
535User-agent: bedrockbo
536Disallow: /
537
538User-agent: CohereBot
539Disallow: /
540
541User-agent: Doubaobot
542Disallow: /
543
544User-agent: ERNIEBot
545Disallow: /
546
547User-agent: GeistHaus-PageFetche
548Disallow: /
549
550User-agent: GoogleAgent-URLContext
551Disallow: /
552
553User-agent: Kimi-SearchBot
554Disallow: /
555
556User-agent: Kimi-User
557Disallow: /
558
559User-agent: KimiBot
560Disallow: /
561
562User-agent: KimiCrawler
563Disallow: /
564
565User-agent: micro-crawl
566Disallow: /
567
568User-agent: MistralAI-Index
569Disallow: /
570
571User-agent: MistralBot
572Disallow: /
573
574User-agent: MoonshotBot
575Disallow: /
576
577User-agent: MoonSpider
578Disallow: /
579
580User-agent: QwenBot
581Disallow: /
582
583User-agent: SalamandraVLM
584Disallow: /
585
586User-agent: Shap-User
587Disallow: /
588
589User-agent: TongyiBot
590Disallow: /
591
592User-agent: UseAI
593Disallow: /
594
595User-agent: xAI-SearchBot
596Disallow: /
597
598User-agent: YandexAdditional
599Disallow: /
600
601User-agent: YandexAdditionalBot
602Disallow: /
603
604User-agent: YiyanBot
605Disallow: /
606
607User-agent: ZenphonyBot
608Disallow: /
609
610User-agent: bedrockbot
611Disallow: /
612
613User-agent: GeistHaus-PageFetcher
614Disallow: /
615
616User-agent: *
617Disallow: /tncms/tracking/
618Disallow: /_services/
619Disallow: /buycanadianbriefing/
620Disallow: /newsletters/books/
621Disallow: /newsletters/business/
622Disallow: /newsletters/evening-headlines/
623Disallow: /newsletters/long-reads/
624Disallow: /newsletters/news/
625Disallow: /newsletters/opinion/
626Disallow: /newsletters/real-estate/
627Disallow: /newsletters/sports/
628Disallow: /newsletters/storyalerts/
629Disallow: /newsletters/success/
630Disallow: /newsletters/unsubscribe/
631Disallow: /newspapers/header/
632Disallow: /registration/
633Disallow: /registration/success/
634Disallow: /search/
635Disallow: /site/advertise/
636Disallow: /static/
637Disallow: /subscribe/3forfree/
638Disallow: /subscribe/amplified/
639Disallow: /subscribe/canoo/
640Disallow: /subscribe/promo/
641Disallow: /subscribe/registrant-onboarding-25/
642Disallow: /subscribe/registrant-onboarding-40/
643Disallow: /subscribe/registrant-onboarding-50/
644Disallow: /subscribe/special/
645Disallow: /subscribe/trial-purchase/
646Disallow: /subscribe/trial-purchase-success/
647Disallow: /subscribe/tryfree/
648Disallow: /subscribe/wb-auto/
649Disallow: /t/
650Disallow: /tapp/
651Disallow: /test/
652Disallow: /test_aplus/
653Disallow: /tncms/gtm/
654Disallow: /tncms/dmp/
655Disallow: /tncms/messaging/
656Disallow: /tncms/tax/
657Disallow: /tncms/openid2/
658Disallow: /tncms/weather/
659Disallow: /tncms/csrf/
660Disallow: /tncms/track/
661Disallow: /tncms/media/
662Disallow: /tncms/webservice/
663Disallow: /tncms/search/
664Disallow: /tncms/counter/
665Disallow: /tncms/openweb/*
666Disallow: /tncms/auth/
667Disallow: /tncms/block/
668Disallow: /tncms/admin/
669Disallow: /tncms/user/
670Disallow: /users/admin/
671Disallow: /users/*/?
672Disallow: /content/tncms/assets/v3/form/
673Disallow: /content/tncms/form/
674Disallow: /content/tncms/
675Disallow: /tncms/disqus/
676Disallow: /marketplace/*action=srch
677Disallow: /tncms/calendar/
678Disallow: /tncms/asset/editorial/http*
679Disallow: /*%7b%7burl%7d%7d
680Disallow: /*%7B%7Burl%7D%7D
681Disallow: /*{{url}}
682Disallow: /content/tncms/live/
683Disallow: /assets/fonts/
684Disallow: /content/dam/
685Disallow: /*li_medium=
686Disallow: /*li_source=
687
688Sitemap: https://www.thestar.com/sitemap.xml
689Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml
690Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1979
691Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1980
692Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1984
693Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1985
694Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1992
695Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1993
696Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1995
697Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1998
698Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=1999
699Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2002
700Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2003
701Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2005
702Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2007
703Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2008
704Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2009
705Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2010
706Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2011
707Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2012
708Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2013
709Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2014
710Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2015
711Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2016
712Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2017
713Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2018
714Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2019
715Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2020
716Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2021
717Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2022
718Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2023
719Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2024
720Sitemap: https://www.thestar.com/tncms/sitemap/editorial.xml?year=2025