Toronto Star / robots.txt
← back to thestar.com · change detected 30d ago (2026-10-08T16:07:06Z) · modified +35 −14
What changed
- Blocked user-agent ‘AgentDataBot’ entirely
- Blocked user-agent ‘BeansLLM-CorpusBot’ entirely
- Blocked user-agent ‘BixelBot’ entirely
- Blocked user-agent ‘DesearchBot’ entirely
- Blocked user-agent ‘Hunyuan’ entirely
- Blocked user-agent ‘ImageBot’ entirely
- Blocked user-agent ‘Keenable-User’ entirely
Diff old (2026-10-08T10:00:34Z) → new (2026-10-08T16:07:06Z)
| @@ -661,6 +661,27 @@ | ||
| 661 | 661 | User-agent: Spider |
| 662 | 662 | Disallow: / |
| 663 | 663 | |
| 664 | + User-agent: AgentDataBot | |
| 665 | + Disallow: / | |
| 666 | + | |
| 667 | + User-agent: BeansLLM-CorpusBot | |
| 668 | + Disallow: / | |
| 669 | + | |
| 670 | + User-agent: BixelBot | |
| 671 | + Disallow: / | |
| 672 | + | |
| 673 | + User-agent: DesearchBot | |
| 674 | + Disallow: / | |
| 675 | + | |
| 676 | + User-agent: Hunyuan | |
| 677 | + Disallow: / | |
| 678 | + | |
| 679 | + User-agent: ImageBot | |
| 680 | + Disallow: / | |
| 681 | + | |
| 682 | + User-agent: Keenable-User | |
| 683 | + Disallow: / | |
| 684 | + | |
| 664 | 685 | User-agent: * |
| 665 | 686 | Disallow: /tncms/tracking/ |
| 666 | 687 | Disallow: /_services/ |
| @@ -696,31 +717,31 @@ | ||
| 696 | 717 | Disallow: /tapp/ |
| 697 | 718 | Disallow: /test/ |
| 698 | 719 | Disallow: /test_aplus/ |
| 699 | - Disallow: /tncms/gtm/ | |
| 700 | - Disallow: /tncms/openweb/* | |
| 701 | - Disallow: /tncms/csrf/ | |
| 720 | + Disallow: /tncms/dmp/ | |
| 721 | + Disallow: /tncms/block/ | |
| 702 | 722 | Disallow: /tncms/openid2/ |
| 703 | - Disallow: /tncms/block/ | |
| 704 | - Disallow: /tncms/track/ | |
| 723 | + Disallow: /tncms/webservice/ | |
| 705 | 724 | Disallow: /tncms/search/ |
| 706 | - Disallow: /tncms/weather/ | |
| 725 | + Disallow: /tncms/tax/ | |
| 707 | 726 | Disallow: /tncms/counter/ |
| 708 | 727 | Disallow: /tncms/admin/ |
| 709 | - Disallow: /tncms/dmp/ | |
| 710 | - Disallow: /tncms/webservice/ | |
| 728 | + Disallow: /tncms/weather/ | |
| 729 | + Disallow: /tncms/openweb/* | |
| 730 | + Disallow: /tncms/track/ | |
| 711 | 731 | Disallow: /tncms/messaging/ |
| 712 | - Disallow: /tncms/tax/ | |
| 732 | + Disallow: /tncms/gtm/ | |
| 733 | + Disallow: /tncms/media/ | |
| 734 | + Disallow: /tncms/csrf/ | |
| 713 | 735 | Disallow: /tncms/auth/ |
| 714 | - Disallow: /tncms/media/ | |
| 715 | 736 | Disallow: /content/tncms/ |
| 716 | - Disallow: /tncms/calendar/ | |
| 717 | - Disallow: /content/tncms/assets/v3/form/ | |
| 718 | - Disallow: /content/tncms/form/ | |
| 719 | - Disallow: /marketplace/*action=srch | |
| 720 | 737 | Disallow: /tncms/user/ |
| 721 | 738 | Disallow: /users/admin/ |
| 722 | 739 | Disallow: /users/*/? |
| 723 | 740 | Disallow: /tncms/disqus/ |
| 741 | + Disallow: /content/tncms/assets/v3/form/ | |
| 742 | + Disallow: /content/tncms/form/ | |
| 743 | + Disallow: /marketplace/*action=srch | |
| 744 | + Disallow: /tncms/calendar/ | |
| 724 | 745 | Disallow: /tncms/asset/editorial/http* |
| 725 | 746 | Disallow: /*%7b%7burl%7d%7d |
| 726 | 747 | Disallow: /*%7B%7Burl%7D%7D |