NeuralCrawl

Toronto Star / robots.txt

← back to thestar.com · change detected 24d ago (2026-07-13T22:52:30Z) · modified +353 −17

What changed

Diff old (2026-07-13T16:47:36Z) → new (2026-07-13T22:52:30Z)

@@ -277,6 +277,342 @@
277277  User-agent: TerraCotta
278278  Disallow: /
279279 
280+ User-agent: ScraperAPI
281+ Disallow: /
282+
283+ User-agent: Scraper-API
284+ Disallow: /
285+
286+ User-agent: ScrapingBee
287+ Disallow: /
288+
289+ User-agent: ScrapingBeeBot
290+ Disallow: /
291+
292+ User-agent: ZenRows
293+ Disallow: /
294+
295+ User-agent: ZenRowsBot
296+ Disallow: /
297+
298+ User-agent: ScrapeGraphAI
299+ Disallow: /
300+
301+ User-agent: scrapegraph
302+ Disallow: /
303+
304+ User-agent: Scrapfly
305+ Disallow: /
306+
307+ User-agent: ScrapflyBot
308+ Disallow: /
309+
310+ User-agent: ScrapflyClient
311+ Disallow: /
312+
313+ User-agent: Zyte
314+ Disallow: /
315+
316+ User-agent: ZyteBot
317+ Disallow: /
318+
319+ User-agent: Scrapinghub
320+ Disallow: /
321+
322+ User-agent: SerpApi
323+ Disallow: /
324+
325+ User-agent: SerpApiBot
326+ Disallow: /
327+
328+ User-agent: WebScraping.AI
329+ Disallow: /
330+
331+ User-agent: WebScrapingAI
332+ Disallow: /
333+
334+ User-agent: webscraping-ai
335+ Disallow: /
336+
337+ User-agent: Scrapeless
338+ Disallow: /
339+
340+ User-agent: ScrapelessBot
341+ Disallow: /
342+
343+ User-agent: Scrapingdog
344+ Disallow: /
345+
346+ User-agent: ScrapingdogBot
347+ Disallow: /
348+
349+ User-agent: Scrapestack
350+ Disallow: /
351+
352+ User-agent: ScrapestackBot
353+ Disallow: /
354+
355+ User-agent: ScaleSERP
356+ Disallow: /
357+
358+ User-agent: SerpWow
359+ Disallow: /
360+
361+ User-agent: Rainforest-API
362+ Disallow: /
363+
364+ User-agent: BlueCart-API
365+ Disallow: /
366+
367+ User-agent: BrightData
368+ Disallow: /
369+
370+ User-agent: Bright Data
371+ Disallow: /
372+
373+ User-agent: BrightBot
374+ Disallow: /
375+
376+ User-agent: Smartproxy
377+ Disallow: /
378+
379+ User-agent: Oxylabs
380+ Disallow: /
381+
382+ User-agent: OxylabsBot
383+ Disallow: /
384+
385+ User-agent: Soax
386+ Disallow: /
387+
388+ User-agent: SoaxBot
389+ Disallow: /
390+
391+ User-agent: SOAX
392+ Disallow: /
393+
394+ User-agent: Nimbleway
395+ Disallow: /
396+
397+ User-agent: NimbleCrawler
398+ Disallow: /
399+
400+ User-agent: NimblewayBot
401+ Disallow: /
402+
403+ User-agent: Froxy
404+ Disallow: /
405+
406+ User-agent: FroxyBot
407+ Disallow: /
408+
409+ User-agent: BrowseAI
410+ Disallow: /
411+
412+ User-agent: Browse-AI
413+ Disallow: /
414+
415+ User-agent: BrowseAIBot
416+ Disallow: /
417+
418+ User-agent: Octoparse
419+ Disallow: /
420+
421+ User-agent: OctoparseBot
422+ Disallow: /
423+
424+ User-agent: Import.io
425+ Disallow: /
426+
427+ User-agent: ImportIO
428+ Disallow: /
429+
430+ User-agent: importio
431+ Disallow: /
432+
433+ User-agent: Kadoa
434+ Disallow: /
435+
436+ User-agent: Kadoabot
437+ Disallow: /
438+
439+ User-agent: KadoaBot
440+ Disallow: /
441+
442+ User-agent: ScrapeHero
443+ Disallow: /
444+
445+ User-agent: Scrape-Hero
446+ Disallow: /
447+
448+ User-agent: ScrapeHeroBot
449+ Disallow: /
450+
451+ User-agent: Grepsr
452+ Disallow: /
453+
454+ User-agent: GrepsrBot
455+ Disallow: /
456+
457+ User-agent: GrepsrCrawler
458+ Disallow: /
459+
460+ User-agent: MrScraper
461+ Disallow: /
462+
463+ User-agent: Mr-Scraper
464+ Disallow: /
465+
466+ User-agent: MrScraperBot
467+ Disallow: /
468+
469+ User-agent: PromptCloud
470+ Disallow: /
471+
472+ User-agent: PromptCloudBot
473+ Disallow: /
474+
475+ User-agent: CrawlNow
476+ Disallow: /
477+
478+ User-agent: CrawlNowBot
479+ Disallow: /
480+
481+ User-agent: ForageAI
482+ Disallow: /
483+
484+ User-agent: Forage-AI
485+ Disallow: /
486+
487+ User-agent: ForageBot
488+ Disallow: /
489+
490+ User-agent: DumplingAI
491+ Disallow: /
492+
493+ User-agent: Dumpling-AI
494+ Disallow: /
495+
496+ User-agent: DumplingBot
497+ Disallow: /
498+
499+ User-agent: spider-cloud
500+ Disallow: /
501+
502+ User-agent: SpiderCloud
503+ Disallow: /
504+
505+ User-agent: Hyperbrowser
506+ Disallow: /
507+
508+ User-agent: HyperbrowserBot
509+ Disallow: /
510+
511+ User-agent: HyperAgent
512+ Disallow: /
513+
514+ User-agent: skrape.ai
515+ Disallow: /
516+
517+ User-agent: SkrapeAI
518+ Disallow: /
519+
520+ User-agent: SkrapeBot
521+ Disallow: /
522+
523+ User-agent: CloudflareBrowserRenderingCrawler
524+ Disallow: /
525+
526+ User-agent: Apify-HTTP-Client
527+ Disallow: /
528+
529+ User-agent: amazon-kendra
530+ Disallow: /
531+
532+ User-agent: amazon-QBusiness
533+ Disallow: /
534+
535+ User-agent: bedrockbo
536+ Disallow: /
537+
538+ User-agent: CohereBot
539+ Disallow: /
540+
541+ User-agent: Doubaobot
542+ Disallow: /
543+
544+ User-agent: ERNIEBot
545+ Disallow: /
546+
547+ User-agent: GeistHaus-PageFetche
548+ Disallow: /
549+
550+ User-agent: GoogleAgent-URLContext
551+ Disallow: /
552+
553+ User-agent: Kimi-SearchBot
554+ Disallow: /
555+
556+ User-agent: Kimi-User
557+ Disallow: /
558+
559+ User-agent: KimiBot
560+ Disallow: /
561+
562+ User-agent: KimiCrawler
563+ Disallow: /
564+
565+ User-agent: micro-crawl
566+ Disallow: /
567+
568+ User-agent: MistralAI-Index
569+ Disallow: /
570+
571+ User-agent: MistralBot
572+ Disallow: /
573+
574+ User-agent: MoonshotBot
575+ Disallow: /
576+
577+ User-agent: MoonSpider
578+ Disallow: /
579+
580+ User-agent: QwenBot
581+ Disallow: /
582+
583+ User-agent: SalamandraVLM
584+ Disallow: /
585+
586+ User-agent: Shap-User
587+ Disallow: /
588+
589+ User-agent: TongyiBot
590+ Disallow: /
591+
592+ User-agent: UseAI
593+ Disallow: /
594+
595+ User-agent: xAI-SearchBot
596+ Disallow: /
597+
598+ User-agent: YandexAdditional
599+ Disallow: /
600+
601+ User-agent: YandexAdditionalBot
602+ Disallow: /
603+
604+ User-agent: YiyanBot
605+ Disallow: /
606+
607+ User-agent: ZenphonyBot
608+ Disallow: /
609+
610+ User-agent: bedrockbot
611+ Disallow: /
612+
613+ User-agent: GeistHaus-PageFetcher
614+ Disallow: /
615+
280616  User-agent: *
281617  Disallow: /tncms/tracking/
282618  Disallow: /_services/
@@ -313,30 +649,30 @@
313649  Disallow: /test/
314650  Disallow: /test_aplus/
315651  Disallow: /tncms/gtm/
652+ Disallow: /tncms/media/
653+ Disallow: /tncms/messaging/
654+ Disallow: /tncms/csrf/
655+ Disallow: /tncms/track/
656+ Disallow: /tncms/auth/
657+ Disallow: /tncms/webservice/
658+ Disallow: /tncms/admin/
659+ Disallow: /tncms/weather/
660+ Disallow: /tncms/block/
661+ Disallow: /tncms/search/
316662  Disallow: /tncms/openweb/*
317- Disallow: /tncms/csrf/
663+ Disallow: /tncms/counter/
664+ Disallow: /tncms/tax/
318665  Disallow: /tncms/openid2/
319- Disallow: /tncms/block/
320- Disallow: /tncms/track/
321- Disallow: /tncms/search/
322- Disallow: /tncms/weather/
323- Disallow: /tncms/counter/
324- Disallow: /tncms/admin/
325666  Disallow: /tncms/dmp/
326- Disallow: /tncms/webservice/
327- Disallow: /tncms/messaging/
328- Disallow: /tncms/tax/
329- Disallow: /tncms/auth/
330- Disallow: /tncms/media/
331- Disallow: /content/tncms/
332- Disallow: /tncms/calendar/
333- Disallow: /content/tncms/assets/v3/form/
334- Disallow: /content/tncms/form/
335- Disallow: /marketplace/*action=srch
336667  Disallow: /tncms/user/
337668  Disallow: /users/admin/
338669  Disallow: /users/*/?
339670  Disallow: /tncms/disqus/
671+ Disallow: /content/tncms/
672+ Disallow: /tncms/calendar/
673+ Disallow: /marketplace/*action=srch
674+ Disallow: /content/tncms/assets/v3/form/
675+ Disallow: /content/tncms/form/
340676  Disallow: /tncms/asset/editorial/http*
341677  Disallow: /*%7b%7burl%7d%7d
342678  Disallow: /*%7B%7Burl%7D%7D