NeuralCrawl

Le Monde / robots.txt snapshot

← back to lemonde.fr · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 12039 bytes · sha256 bed9b205ddee9529 · raw

final URL: https://www.lemonde.fr/robots.txt

1# 16/08/2019
2# Il est interdit d'utiliser des robots d'indexation Web ou d'autres méthodes automatiques de feuilletage ou de navigation sur ce site Web.
3# Nous interdisons de crawler notre site Web en utilisant un agent d'utilisateur volé qui ne correspond pas à votre identité.
4# « Violation du droit du producteur de base de données - article L 342-1 et suivant le Code de la propriété intellectuelle ».
5# Nous vous invitons à nous contacter pour contracter une licence d'utilisation. Seuls les partenaires sont habilités à utiliser nos contenus pour un usage autre que strictement individuel.
6User-agent: *
7Allow: /ws/1/live/*
8Allow: /ws/1/related_content/*
9Disallow: /ajax/
10Disallow: /ajah/
11Disallow: /api/
12Disallow: /beta
13Disallow: /element/commun/afficher/
14Disallow: /petites-annonces/
15Disallow: /qui-sommes-nous/
16Disallow: /txt/
17Disallow: /verification/source/*
18Disallow: /noscript/
19Disallow: /ws/*
20Disallow: /lemonde-beta/*
21Disallow: /_rprt/*
22Disallow: /layout/*
23Disallow: /cgi-bin/*
24Disallow: /envoyer-par-email/*
25Disallow: /lmdgft/*
26Disallow: /article-offert/*
27Disallow: /*?s=43260*
28Disallow: /*?contributions
29Disallow: */reactions/
30Disallow: */mmpub/
31# WordPress
32Disallow: /blog/*/wp-admin/
33Disallow: /blog/*/wp-includes/
34Disallow: /blog/*/wp-content/plugins/
35Disallow: /blog/*/wp-content/themes/
36Disallow: /blog/*/wp-login.php
37Disallow: /blog/*/wp-register.php
38Disallow: /blog/*/author/admin/
39# Recherche
40Disallow: /recherche/?*search_keywords=*
41# Sitemaps
42Sitemap: https://www.lemonde.fr/sitemap_news.xml
43Sitemap: https://www.lemonde.fr/sitemap_index.xml
44# Sitemaps EN
45Sitemap: https://www.lemonde.fr/en/sitemap_news.xml
46Sitemap: https://www.lemonde.fr/en/sitemap_index.xml
47
48User-agent: Googlebot-Image
49Allow: /image/
50
51User-agent: Googlebot-News
52Disallow: /archives/
53
54# Robots exclus de toute indexation.
55User-agent: Cision
56Disallow: /
57
58User-agent: Talkwater
59Disallow: /
60
61User-agent: Jetbot
62Disallow: /
63
64User-agent: kbcrawl
65Disallow: /
66
67User-agent: Newzbin
68Disallow: /
69
70User-agent: Qwam content intelligence
71Disallow: /
72
73User-agent: Youmag
74Disallow: /
75
76User-agent: Synthesio
77Disallow: /
78
79User-agent: trendybuzz
80Disallow: /
81
82User-agent: scoop.it
83Disallow: /
84
85User-agent: linkfluence
86Disallow: /
87
88User-agent: grub-client
89Disallow: /
90
91User-agent: ia_archiver-web.archive.org
92Allow: /$
93Disallow: /*
94
95User-agent: k2spider
96Disallow: /
97
98User-agent: libwww
99Disallow: /
100
101User-agent: wget
102Disallow: /
103
104User-agent: 5erue
105Disallow: /
106
107User-agent: adequat
108Disallow: /
109
110User-agent: adequat-systems
111Disallow: /
112
113User-agent: coexel
114Disallow: /
115
116User-agent: leadbox
117Disallow: /
118
119User-agent: mention
120Disallow: /
121
122User-agent: mytwip
123Disallow: /
124
125User-agent: opinion-tracker
126Disallow: /
127
128User-agent: proxem
129Disallow: /
130
131User-agent: score3
132Disallow: /
133
134User-agent: vecteurplus
135Disallow: /
136
137User-agent: verticalsearch
138Disallow: /
139
140User-agent: vsw
141Disallow: /
142
143User-agent: winello
144Disallow: /
145
146User-agent: Fetch
147Disallow: /
148
149User-agent: infoseek
150Disallow: /
151
152User-agent: MSIECrawler
153Disallow: /
154
155User-agent: Offline Explorer
156Disallow: /
157
158User-agent: sitecheck.internetseer.com
159Disallow: /
160
161User-agent: Teleport
162Disallow: /
163
164User-agent: TeleportPro
165Disallow: /
166
167User-agent: WebCopier
168Disallow: /
169
170User-agent: WebStripper
171Disallow: /
172
173User-agent: Zealbot
174Disallow: /
175
176User-agent: asknread.com
177Disallow: /
178
179User-agent: omgilibot
180Disallow: /
181
182User-agent: omgili
183Disallow: /
184
185User-agent: CCBot
186Disallow: /
187
188User-agent: Google-Extended
189Disallow: /
190
191User-agent: anthropic-ai
192Disallow: /
193
194User-agent: Claude-Web
195Disallow: /
196
197User-agent: ClaudeBot
198Disallow: /
199
200User-agent: Applebot-Extended
201Disallow: /
202
203User-agent: Webzio-Extended
204Disallow: /
205
206User-agent: Amazonbot
207Disallow: /
208
209User-agent: Timpibot
210Disallow: /
211
212User-agent: AI2Bot
213Disallow: /
214
215User-agent: cohere-training-data-crawler
216Disallow: /
217
218User-agent: DuckAssistBot
219Disallow: /
220
221User-agent: Kangaroo Bot
222Disallow: /
223
224User-agent: PanguBot
225Disallow: /
226
227User-agent: MistralAI-User
228Disallow: /
229
230# Cas particulier pour les bots de Facebook
231User-agent: facebookbot
232Disallow: /
233
234User-agent: FacebookBot
235Disallow: /
236
237# LISTEROBOTS1802
238User-agent: 5emeRue
239Disallow: /
240
241User-agent: ACQUIRE MEDIA
242Disallow: /
243
244User-agent: ACTIV Financial (CME Group)
245Disallow: /
246
247User-agent: AlphaSense
248Disallow: /
249
250User-agent: AmiSoftware
251Disallow: /
252
253User-agent: archive.org_bot
254Disallow: /
255
256User-agent: Archive-It
257Disallow: /
258
259User-agent: ArgClrInt
260Disallow: /
261
262User-agent: Ask n read
263Disallow: /
264
265User-agent: Augure
266Disallow: /
267
268User-agent: auramundi
269Disallow: /
270
271User-agent: AwarioRssBot
272Disallow: /
273
274User-agent: AwarioSmartBot
275Disallow: /
276
277User-agent: Barchart.com
278Disallow: /
279
280User-agent: BattleFin
281Disallow: /
282
283User-agent: Bernin IT
284Disallow: /
285
286User-agent: Blackboard Safeassign
287Disallow: /
288
289User-agent: BLP_bbot
290Disallow: /
291
292User-agent: bluematrix
293Disallow: /
294
295User-agent: Brandwatch
296Disallow: /
297
298User-agent: Briefcase.news
299Disallow: /
300
301User-agent: Buck
302Disallow: /
303
304User-agent: Bytespider
305Disallow: /
306
307User-agent: CCBo
308Disallow: /
309
310User-agent: CikisiBot
311Disallow: /
312
313User-agent: Coexel
314Disallow: /
315
316User-agent: cohere-ai
317Disallow: /
318
319User-agent: Comtex News Network
320Disallow: /
321
322User-agent: ConveraCrawler
323Disallow: /
324
325User-agent: Copyright Licensing Agency
326Disallow: /
327
328User-agent: Corporama
329Disallow: /
330
331User-agent: D&B Hoovers
332Disallow: /
333
334User-agent: Data Expression
335Disallow: /
336
337User-agent: Data Observer
338Disallow: /
339
340User-agent: Dataminr
341Disallow: /
342
343User-agent: Dealogic
344Disallow: /
345
346User-agent: Diffbot
347Disallow: /
348
349User-agent: Digimind
350Disallow: /
351
352User-agent: DirectFN
353Disallow: /
354
355User-agent: Dun & Bradstreet
356Disallow: /
357
358User-agent: Dun & Bradstreet - D&B ESG Intelligence
359Disallow: /
360
361User-agent: Dun & Bradstreet Data Marketplace
362Disallow: /
363
364User-agent: Eagle Alpha
365Disallow: /
366
367User-agent: ecoresearch
368Disallow: /
369
370User-agent: ellisphere
371Disallow: /
372
373User-agent: FactSet
374Disallow: /
375
376User-agent: FeedCheck
377Disallow: /
378
379User-agent: FeedReader
380Disallow: /
381
382User-agent: Feedspot
383Disallow: /
384
385User-agent: Fitch Solutions
386Disallow: /
387
388User-agent: Founder Apabi
389Disallow: /
390
391User-agent: Freshbot
392Disallow: /
393
394User-agent: FriendlyCrawler
395Disallow: /
396
397User-agent: Gnowit
398Disallow: /
399
400User-agent: GnowitNewsbot
401Disallow: /
402
403User-agent: Ground News
404Disallow: /
405
406User-agent: ia_archiver
407Disallow: /
408
409User-agent: ICE Connect Desktop Solution
410Disallow: /
411
412User-agent: ICE Data Services
413Disallow: /
414
415User-agent: IHS Markit
416Disallow: /
417
418User-agent: ImageSift
419Disallow: /
420
421User-agent: InMédia Technologies
422Disallow: /
423
424User-agent: Innguma
425Disallow: /
426
427User-agent: Inoreader
428Disallow: /
429
430User-agent: ISI Emerging Markets
431Disallow: /
432
433User-agent: KB Crawl SAS
434Disallow: /
435
436User-agent: Knowings
437Disallow: /
438
439User-agent: Koyfin
440Disallow: /
441
442User-agent: Launchmetrics
443Disallow: /
444
445User-agent: LexisNexis
446Disallow: /
447
448User-agent: Liana
449Disallow: /
450
451User-agent: magpie-crawler
452Disallow: /
453
454User-agent: Make/production
455Disallow: /
456
457User-agent: MarketResearch.com
458Disallow: /
459
460User-agent: MarketWatch
461Disallow: /
462
463User-agent: MarketWise
464Disallow: /
465
466User-agent: Markit Digital
467Disallow: /
468
469User-agent: Mediatoolkitbot
470Disallow: /
471
472User-agent: moduleQ
473Disallow: /
474
475User-agent: MONITIO
476Disallow: /
477
478User-agent: MoodleBot
479Disallow: /
480
481User-agent: Moody's
482Disallow: /
483
484User-agent: Moreover
485Disallow: /
486
487User-agent: MORNINGSTAR
488Disallow: /
489
490User-agent: MuckRack
491Disallow: /
492
493User-agent: Netvibes
494Disallow: /
495
496User-agent: news-api.org
497Disallow: /
498
499User-agent: Newslitbot
500Disallow: /
501
502User-agent: NewsNow
503Disallow: /
504
505User-agent: Northern Light
506Disallow: /
507
508User-agent: Opinion-tracker
509Disallow: /
510
511User-agent: Orbis
512Disallow: /
513
514User-agent: Opoint
515Disallow: /
516
517User-agent: Paqlebot
518Disallow: /
519
520User-agent: Press Monitor Europe
521Disallow: /
522
523User-agent: PressEngineBot D
524Disallow: /
525
526User-agent: PriberamBot
527Disallow: /
528
529User-agent: QuoteMedia
530Disallow: /
531
532User-agent: QWAM CONTENT INTELLIGENCE
533Disallow: /
534
535User-agent: RankurBot
536Disallow: /
537
538User-agent: RavenPack
539Disallow: /
540
541User-agent: ReportLinker
542Disallow: /
543
544User-agent: Research & Markets
545Disallow: /
546
547User-agent: S&P Capital IQ
548Disallow: /
549
550User-agent: S&P Global Market Intelligence
551Disallow: /
552
553User-agent: S&P Global Marketplace
554Disallow: /
555
556User-agent: scoopit
557Disallow: /
558
559User-agent: scoopit-crawler
560Disallow: /
561
562User-agent: scpitspi-rs
563Disallow: /
564
565User-agent: semantic-visions.com
566Disallow: /
567
568User-agent: semantic-visions.com crawler
569Disallow: /
570
571User-agent: SemrushBot
572Disallow: /
573
574User-agent: SentiBot
575Disallow: /
576
577User-agent: SentiOne
578Disallow: /
579
580User-agent: Signal Insights
581Disallow: /
582
583User-agent: Sindup
584Disallow: /
585
586User-agent: smartkarma
587Disallow: /
588
589User-agent: Sociallymap
590Disallow: /
591
592User-agent: spotter
593Disallow: /
594
595User-agent: squirrobot
596Disallow: /
597
598User-agent: Statista
599Disallow: /
600
601User-agent: STOCKBOARD
602Disallow: /
603
604User-agent: telpress.it
605Disallow: /
606
607User-agent: Thomson Reuters
608Disallow: /
609
610User-agent: Thomson Reuters WestLaw
611Disallow: /
612
613User-agent: TraderPlanet
614Disallow: /
615
616User-agent: Tradingcharts.com
617Disallow: /
618
619User-agent: TradingView
620Disallow: /
621
622User-agent: Transform Your API
623Disallow: /
624
625User-agent: trendeo
626Disallow: /
627
628User-agent: trendictionbot
629Disallow: /
630
631User-agent: TurnitinBot
632Disallow: /
633
634User-agent: um-FC
635Disallow: /
636
637User-agent: um-IC
638Disallow: /
639
640User-agent: um-LN
641Disallow: /
642
643User-agent: UptimeRobot
644Disallow: /
645
646User-agent: Vable Ltd
647Disallow: /
648
649User-agent: VISIBRAIN
650Disallow: /
651
652User-agent: webzio
653Disallow: /
654
655User-agent: YaK
656Disallow: /
657
658User-agent: YCHARTS
659Disallow: /
660
661User-agent: YouBot
662Disallow: /
663
664User-agent: Zite
665Disallow: /
666
667User-agent: Cotoyogi
668Disallow: /
669
670User-agent: Datenbank Crawler
671Disallow: /
672
673User-agent: Devin
674Disallow: /
675
676User-agent: Factset_spyderbot
677Disallow: /
678
679User-agent: ICC-Crawler
680Disallow: /
681
682User-agent: PetalBot
683Disallow: /
684
685User-agent: SemrushBot-OCOB
686Disallow: /
687
688User-agent: AddSearchBot
689Disallow: /
690
691User-agent: bigsur.ai
692Disallow: /
693
694User-agent: Claude-SearchBot
695Disallow: /
696
697User-agent: Claude-User
698Disallow: /
699
700User-agent: CloudVertexBot
701Disallow: /
702
703User-agent: Gemini-Deep-Research
704Disallow: /
705
706User-agent: GoogleOther
707Disallow: /
708
709User-agent: LinerBot
710Disallow: /
711
712User-agent: netEstate Imprint Crawler
713Disallow: /
714
715User-agent: QualifiedBot
716Disallow: /
717
718User-agent: AI2Bot-DeepResearchEval
719Disallow: /
720
721User-agent: Ai2Bot-Dolma
722Disallow: /
723
724User-agent: Anomura
725Disallow: /
726
727User-agent: ApifyBot
728Disallow: /
729
730User-agent: ApifyWebsiteContentCrawler
731Disallow: /
732
733User-agent: Aranet-SearchBot
734Disallow: /
735
736User-agent: atlassian-bot
737Disallow: /
738
739User-agent: AzureAI-SearchBot
740Disallow: /
741
742User-agent: Bravebot
743Disallow: /
744
745User-agent: Channel3Bot
746Disallow: /
747
748User-agent: ChatGLM-Spider
749Disallow: /
750
751User-agent: Cloudflare-AutoRAG
752Disallow: /
753
754User-agent: Crawl4AI
755Disallow: /
756
757User-agent: DeepSeekBot
758Disallow: /
759
760User-agent: ExaBot
761Disallow: /
762
763User-agent: FirecrawlAgent
764Disallow: /
765
766User-agent: Google-CloudVertexBot
767Disallow: /
768
769User-agent: Google-NotebookLM
770Disallow: /
771
772User-agent: iAskBot
773Disallow: /
774
775User-agent: iaskspider
776Disallow: /
777
778User-agent: imageSpider
779Disallow: /
780
781User-agent: kagi-fetcher
782Disallow: /
783
784User-agent: KlaviyoAIBot
785Disallow: /
786
787User-agent: KunatoCrawler
788Disallow: /
789
790User-agent: laion-huggingface-processor
791Disallow: /
792
793User-agent: LCC
794Disallow: /
795
796User-agent: LinkupBot
797Disallow: /
798
799User-agent: meta-webindexer
800Disallow: /
801
802User-agent: PhindBot
803Disallow: /
804
805User-agent: Poggio-Citations
806Disallow: /
807
808User-agent: SBIntuitionsBot
809Disallow: /
810
811User-agent: Spider
812Disallow: /
813
814User-agent: TavilyBot
815Disallow: /
816
817User-agent: TerraCotta
818Disallow: /
819
820User-agent: VelenPublicWebCrawler
821Disallow: /
822
823User-agent: webzio-extended
824Disallow: /
825
826User-agent: WRTNBot
827Disallow: /
828
829User-agent: ZanistaBot
830Disallow: /
831
832User-agent: AmazonBuyForMe
833Disallow: /
834
835User-agent: Google-Agent
836Disallow: /
837
838User-agent: GoogleAgent-Mariner
839Disallow: /
840
841User-agent: Manus-User
842Disallow: /
843
844User-agent: NovaAct
845Disallow: /
846
847User-agent: Shap-User
848Disallow: /
849
850User-agent: TwinAgent
851Disallow: /
852
853User-agent: ZenphonyBot
854Disallow: /
855
856User-agent: Brightbot
857Disallow: /
858
859User-agent: HenkBot
860Disallow: /
861
862User-agent: ShapBot
863Disallow: /
864
865User-agent: Terra Cotta
866Disallow: /