Le Monde / robots.txt
← back to lemonde.fr · change detected 1mo ago (2026-07-01T18:33:05Z) · modified +39 −0
What changed
- Blocked user-agent ‘CohereBot’ entirely
- Blocked user-agent ‘CragCrawler’ entirely
- Blocked user-agent ‘GeistHaus-PageFetcher’ entirely
- Blocked user-agent ‘GoogleAgent-URLContext’ entirely
- Blocked user-agent ‘Kimi-SearchBot’ entirely
- Blocked user-agent ‘Kimi-User’ entirely
- Blocked user-agent ‘KimiBot’ entirely
- Blocked user-agent ‘MistralAI-Index’ entirely
- Blocked user-agent ‘MistralBot’ entirely
- Blocked user-agent ‘Querit-SearchBot’ entirely
- Blocked user-agent ‘QueritBot’ entirely
- Blocked user-agent ‘UseAI’ entirely
- Blocked user-agent ‘xAI-SearchBot’ entirely
Diff old (2026-06-20T01:10:30Z) → new (2026-07-01T18:33:05Z)
| @@ -864,3 +864,42 @@ | ||
| 864 | 864 | |
| 865 | 865 | User-agent: Terra Cotta |
| 866 | 866 | Disallow: / |
| 867 | + | |
| 868 | + User-agent: CohereBot | |
| 869 | + Disallow: / | |
| 870 | + | |
| 871 | + User-agent: CragCrawler | |
| 872 | + Disallow: / | |
| 873 | + | |
| 874 | + User-agent: GeistHaus-PageFetcher | |
| 875 | + Disallow: / | |
| 876 | + | |
| 877 | + User-agent: GoogleAgent-URLContext | |
| 878 | + Disallow: / | |
| 879 | + | |
| 880 | + User-agent: Kimi-SearchBot | |
| 881 | + Disallow: / | |
| 882 | + | |
| 883 | + User-agent: Kimi-User | |
| 884 | + Disallow: / | |
| 885 | + | |
| 886 | + User-agent: KimiBot | |
| 887 | + Disallow: / | |
| 888 | + | |
| 889 | + User-agent: MistralAI-Index | |
| 890 | + Disallow: / | |
| 891 | + | |
| 892 | + User-agent: MistralBot | |
| 893 | + Disallow: / | |
| 894 | + | |
| 895 | + User-agent: Querit-SearchBot | |
| 896 | + Disallow: / | |
| 897 | + | |
| 898 | + User-agent: QueritBot | |
| 899 | + Disallow: / | |
| 900 | + | |
| 901 | + User-agent: UseAI | |
| 902 | + Disallow: / | |
| 903 | + | |
| 904 | + User-agent: xAI-SearchBot | |
| 905 | + Disallow: / |