Toronto Star / robots.txt
← back to thestar.com · change detected 2mo ago (2026-09-01T15:59:14Z) · modified +45 −0
What changed
- Blocked user-agent ‘AIWebIndex’ entirely
- Blocked user-agent ‘AIWebIndex-Agent’ entirely
- Blocked user-agent ‘CragCrawler’ entirely
- Blocked user-agent ‘Crawlspace’ entirely
- Blocked user-agent ‘Datenbank Crawler’ entirely
- Blocked user-agent ‘ExaSearchBot’ entirely
- Blocked user-agent ‘HenkBot’ entirely
- Blocked user-agent ‘Mozilla-Tabstack’ entirely
- Blocked user-agent ‘netEstate Imprint Crawler’ entirely
- Blocked user-agent ‘Querit-SearchBot’ entirely
- Blocked user-agent ‘QueritBot’ entirely
- Blocked user-agent ‘Reflectionbot’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- Blocked user-agent ‘TalarionSearchBot’ entirely
- Blocked user-agent ‘Terra Cotta’ entirely
Diff old (2026-09-01T09:53:30Z) → new (2026-09-01T15:59:14Z)
| @@ -613,6 +613,51 @@ | ||
| 613 | 613 | User-agent: GeistHaus-PageFetcher |
| 614 | 614 | Disallow: / |
| 615 | 615 | |
| 616 | + User-agent: AIWebIndex | |
| 617 | + Disallow: / | |
| 618 | + | |
| 619 | + User-agent: AIWebIndex-Agent | |
| 620 | + Disallow: / | |
| 621 | + | |
| 622 | + User-agent: CragCrawler | |
| 623 | + Disallow: / | |
| 624 | + | |
| 625 | + User-agent: Crawlspace | |
| 626 | + Disallow: / | |
| 627 | + | |
| 628 | + User-agent: Datenbank Crawler | |
| 629 | + Disallow: / | |
| 630 | + | |
| 631 | + User-agent: ExaSearchBot | |
| 632 | + Disallow: / | |
| 633 | + | |
| 634 | + User-agent: HenkBot | |
| 635 | + Disallow: / | |
| 636 | + | |
| 637 | + User-agent: Mozilla-Tabstack | |
| 638 | + Disallow: / | |
| 639 | + | |
| 640 | + User-agent: netEstate Imprint Crawler | |
| 641 | + Disallow: / | |
| 642 | + | |
| 643 | + User-agent: Querit-SearchBot | |
| 644 | + Disallow: / | |
| 645 | + | |
| 646 | + User-agent: QueritBot | |
| 647 | + Disallow: / | |
| 648 | + | |
| 649 | + User-agent: Reflectionbot | |
| 650 | + Disallow: / | |
| 651 | + | |
| 652 | + User-agent: ShapBot | |
| 653 | + Disallow: / | |
| 654 | + | |
| 655 | + User-agent: TalarionSearchBot | |
| 656 | + Disallow: / | |
| 657 | + | |
| 658 | + User-agent: Terra Cotta | |
| 659 | + Disallow: / | |
| 660 | + | |
| 616 | 661 | User-agent: * |
| 617 | 662 | Disallow: /tncms/tracking/ |
| 618 | 663 | Disallow: /_services/ |