NeuralCrawl

Le Monde / robots.txt

← back to lemonde.fr · change detected 1d ago (2026-11-05T13:47:51Z) · modified +137 −3

What changed

Diff old (2026-07-16T11:40:45Z) → new (2026-11-05T13:47:51Z)

@@ -47,6 +47,14 @@
4747 
4848  User-agent: Googlebot-Image
4949  Allow: /image/
50+
51+ User-agent: Google-Extended
52+ Allow: /innovation-et-mobilite/
53+ Allow: /les-conseils-pour-entreprendre/
54+ Allow: /mutuelle-sante-bien-choisir-sa-couverture/
55+ Allow: /prevoir-demain/
56+ Allow: /de-l-impact-aux-responsabilites/
57+ Disallow: /
5058 
5159  User-agent: Googlebot-News
5260  Disallow: /archives/
@@ -183,9 +191,6 @@
183191  Disallow: /
184192 
185193  User-agent: CCBot
186- Disallow: /
187-
188- User-agent: Google-Extended
189194  Disallow: /
190195 
191196  User-agent: anthropic-ai
@@ -957,3 +962,132 @@
957962 
958963  User-agent: YiyanBot
959964  Disallow: /
965+
966+ User-agent: AgentDataBot
967+ Disallow: /
968+
969+ User-agent: AGI Agent
970+ Disallow: /
971+
972+ User-agent: AIWebIndex-Agent
973+ Disallow: /
974+
975+ User-agent: AndiAgent
976+ Disallow: /
977+
978+ User-agent: artsdata-crawler
979+ Disallow: /
980+
981+ User-agent: BeansLLM-CorpusBot
982+ Disallow: /
983+
984+ User-agent: BixelBot
985+ Disallow: /
986+
987+ User-agent: CartAI Agentic Commerce
988+ Disallow: /
989+
990+ User-agent: CledaraBot
991+ Disallow: /
992+
993+ User-agent: Cloudflare-AI-Search
994+ Disallow: /
995+
996+ User-agent: Cloudflare-AI-Search-External
997+ Disallow: /
998+
999+ User-agent: CloudflareBrowserRenderingCrawler
1000+ Disallow: /
1001+
1002+ User-agent: DesearchBot
1003+ Disallow: /
1004+
1005+ User-agent: Dola-SearchBot
1006+ Disallow: /
1007+
1008+ User-agent: Dot
1009+ Disallow: /
1010+
1011+ User-agent: ElevenlabsBot
1012+ Disallow: /
1013+
1014+ User-agent: ElevenlabsCrawler
1015+ Disallow: /
1016+
1017+ User-agent: ExaSearchBot
1018+ Disallow: /
1019+
1020+ User-agent: ExaUser
1021+ Disallow: /
1022+
1023+ User-agent: Grok Bot
1024+ Disallow: /
1025+
1026+ User-agent: grok-agent
1027+ Disallow: /
1028+
1029+ User-agent: GrokBot
1030+ Disallow: /
1031+
1032+ User-agent: HarkBot
1033+ Disallow: /
1034+
1035+ User-agent: Hunyuan
1036+ Disallow: /
1037+
1038+ User-agent: ImageBot
1039+ Disallow: /
1040+
1041+ User-agent: Instinct
1042+ Disallow: /
1043+
1044+ User-agent: Keenable-User
1045+ Disallow: /
1046+
1047+ User-agent: KeenableBot
1048+ Disallow: /
1049+
1050+ User-agent: Kimi-Agent
1051+ Disallow: /
1052+
1053+ User-agent: Muse
1054+ Disallow: /
1055+
1056+ User-agent: Payhawk Invoice Fetching Agent
1057+ Disallow: /
1058+
1059+ User-agent: Quartr Crawler
1060+ Disallow: /
1061+
1062+ User-agent: recraft-image-dataset
1063+ Disallow: /
1064+
1065+ User-agent: ReflectionBot
1066+ Disallow: /
1067+
1068+ User-agent: ScryBot
1069+ Disallow: /
1070+
1071+ User-agent: SofyaBot
1072+ Disallow: /
1073+
1074+ User-agent: SomethinggBot
1075+ Disallow: /
1076+
1077+ User-agent: TalarionFrontierCrawler
1078+ Disallow: /
1079+
1080+ User-agent: TalarionSearchBot
1081+ Disallow: /
1082+
1083+ User-agent: xAI-Grok
1084+ Disallow: /
1085+
1086+ User-agent: YepBot
1087+ Disallow: /
1088+
1089+ User-agent: YiBot
1090+ Disallow: /
1091+
1092+ User-agent: Claude-Agent
1093+ Disallow: /