NeuralCrawl

Le Figaro / robots.txt snapshot

← back to lefigaro.fr · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 10937 bytes · sha256 3951609a5054ebb7 · raw

final URL: https://www.lefigaro.fr/robots.txt

1# Toute utilisation de nos contenus protégés autre qu'un usage strictement individuel (incluant notamment l'entraînement des grands modèles de langage (LLM),
2# l'entraînement des outils d'intelligence artificielle, la veille web ou le media monitoring) est conditionnée à la conclusion d'une licence d'utilisation
3# avec la SOCIETE DU FIGARO. Nous vous invitons à cette fin à contacter [email protected].
4#
5# Toute utilisation non-autorisée de nos contenus protégés est constitutive d'une contrefaçon de droit d'auteur et/ou du droit de producteur
6# de base de données et susceptible d'être poursuivie.
7#
8# Il est interdit de crawler notre site web en utilisant un agent d'utilisateur (user agent) volé qui ne correspond pas à votre identité.
9# L'utilisation des robots d'indexation web ou d'autres méthodes automatiques de feuilletage ou de navigation sur ce site Web n'est pas autorisée.
10
11User-agent: *
12Disallow: /async/
13Disallow: /*?noheader*
14Disallow: /*?*&noheader*
15Disallow: /*?sfdebug*
16Disallow: /*?*&sfdebug*
17Disallow: /brouillon/
18Disallow: /proxy/
19Disallow: /synsearch/
20Disallow: /a-savoir-en-france/
21Disallow: /vos-questions
22
23#LLM
24
25User-agent: ChatGPT-User
26Allow: /voyages
27Allow: /culture
28Allow: /style
29Allow: /bons-plans
30Allow: /elections/resultats
31Disallow: /
32
33User-agent: Claude-SearchBot
34Allow: /voyages
35Allow: /bons-plans
36Allow: /elections/resultats
37Disallow: /
38
39User-agent: Claude-Web
40Allow: /voyages
41Allow: /bons-plans
42Allow: /elections/resultats
43Disallow: /
44
45User-agent: OAI-SearchBot
46Allow: /voyages
47Allow: /culture
48Allow: /style
49Allow: /bons-plans
50Allow: /elections/resultats
51Disallow: /
52
53User-agent: Claude-User
54Allow: /voyages
55Allow: /bons-plans
56Allow: /elections/resultats
57Disallow: /
58
59User-agent: ChatGPT Agent
60Allow: /voyages
61Allow: /bons-plans
62Allow: /elections/resultats
63Disallow: /
64
65User-agent: GoogleAgent-Mariner
66Allow: /voyages
67Allow: /bons-plans
68Allow: /elections/resultats
69Disallow: /
70
71User-agent: Google-NotebookLM
72Allow: /voyages
73Allow: /bons-plans
74Allow: /elections/resultats
75Disallow: /
76
77User-agent: Google-CloudVertexBot
78Allow: /voyages
79Allow: /bons-plans
80Allow: /elections/resultats
81Disallow: /
82
83User-agent: MistralAI-User
84Allow: /voyages
85Allow: /bons-plans
86Allow: /elections/resultats
87Disallow: /
88
89User-agent: Gemini-Deep-Research
90Allow: /voyages
91Allow: /bons-plans
92Allow: /elections/resultats
93Disallow: /
94
95User-agent: CloudVertexBot
96Disallow: /
97
98User-agent: anthropic-ai
99Disallow: /
100
101User-agent: Bytespider
102Disallow: /
103
104User-agent: CCBot
105Disallow: /
106
107User-agent: ClaudeBot
108Disallow: /
109
110User-agent: cohere-ai
111Disallow: /
112
113User-agent: GPTBot
114Disallow: /
115
116User-agent: Google-Extended
117Disallow: /
118
119#Crawler
120
121User-agent: Alexibot
122Disallow: /
123
124User-agent: AlvinetSpider
125Disallow: /
126
127User-agent: AmiSoftware
128Disallow: /
129
130User-agent: Antenne Hatena
131Disallow: /
132
133User-agent: ApifyBot
134Disallow: /
135
136User-agent: ApifyWebsiteContentCrawler
137Disallow: /
138
139User-agent: ApocalXExplorerBot
140Disallow: /
141
142User-agent: Argus
143Disallow: /
144
145User-agent: Ask n read
146Disallow: /
147
148User-agent: asknread.com
149Disallow: /
150
151User-agent: asterias
152Disallow: /
153
154User-agent: BlowFish/1.0
155Disallow: /
156
157User-agent: BotALot
158Disallow: /
159
160User-agent: Brightbot
161Disallow: /
162
163User-agent: BuiltBotTough
164Disallow: /
165
166User-agent: Bullseye/1.0
167Disallow: /
168
169User-agent: BunnySlippers
170Disallow: /
171
172User-agent: Cegbfeieh
173Disallow: /
174
175User-agent: CheeseBot
176Disallow: /
177
178User-agent: ConveraCrawler
179Disallow: /
180
181User-agent: cosmos
182Disallow: /
183
184User-agent: Crescent
185Disallow: /
186
187User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
188Disallow: /
189
190User-agent: cydralspider
191Disallow: /
192
193User-agent: Diffbot
194Disallow: /
195
196User-agent: Diffbot-User
197Disallow: /
198
199User-agent: DISCo Pump 3.1
200Disallow: /
201
202User-agent: DittoSpyder
203Disallow: /
204
205User-agent: EroCrawler
206Disallow: /
207
208User-agent: eureka
209Disallow: /
210
211User-agent: ExaBot
212Disallow: /
213
214User-agent: Explore
215Disallow: /
216
217User-agent: Fetch
218Disallow: /
219
220User-agent: FirecrawlAgent
221Disallow: /
222
223User-agent: Flamingo_SearchEngine
224Disallow: /
225
226User-agent: Foobot
227Disallow: /
228
229User-agent: gammaSpider
230Disallow: /
231
232User-agent: grub-client
233Disallow: /
234
235User-agent: hloader
236Disallow: /
237
238User-agent: httplib
239Disallow: /
240
241User-agent: humanlinks
242Disallow: /
243
244User-agent: ia_archiver
245Allow: /$
246Disallow: /
247
248User-agent: ia_archiver-web.archive.org
249Allow: /$
250Disallow: /
251
252User-agent: indexer
253Disallow: /
254
255User-agent: InfoNaviRobot
256Disallow: /
257
258User-agent: infoseek
259Disallow: /
260
261User-agent: JennyBot
262Disallow: /
263
264User-agent: Jetbot
265Disallow: /
266
267User-agent: JikeSpider
268Disallow: /
269
270User-agent: k2spider
271Disallow: /
272
273User-agent: Kenjin Spider
274Disallow: /
275
276User-agent: larbin
277Disallow: /
278
279User-agent: LexiBot
280Disallow: /
281
282User-agent: libWeb/clsHTTP
283Disallow: /
284
285User-agent: libwww
286Disallow: /
287
288User-agent: linko
289Disallow: /
290
291User-agent: LinkScan/8.1a Unix
292Disallow: /
293
294User-agent: LinkWalker
295Disallow: /
296
297User-agent: lwp-trivial
298Disallow: /
299
300User-agent: lwp-trivial/1.34
301Disallow: /
302
303User-agent: Mata Hari
304Disallow: /
305
306User-agent: Mister PiX
307Disallow: /
308
309User-agent: MLBot
310Disallow: /
311
312User-agent: moget
313Disallow: /
314
315User-agent: moget/2.1
316Disallow: /
317
318User-agent: Naverbot
319Disallow: /
320
321User-agent: NetAttache
322Disallow: /
323
324User-agent: Newzbin
325Disallow: /
326
327User-agent: NICErsPRO
328Disallow: /
329
330User-agent: NPBot
331Disallow: /
332
333User-agent: ObjectsSearch
334Disallow: /
335
336User-agent: omgili
337Disallow: /
338
339User-agent: omgilibot
340Disallow: /
341
342User-agent: Openfind
343Disallow: /
344
345User-agent: OpenindexSpider
346Disallow: /
347
348User-agent: Pimptrain
349Disallow: /
350
351User-agent: ProPowerBot/2.14
352Disallow: /
353
354User-agent: ProWebWalker
355Disallow: /
356
357User-agent: psbot
358Disallow: /
359
360User-agent: QuepasaCreep
361Disallow: /
362
363User-agent: QueryN Metasearch
364Disallow: /
365
366User-agent: Raven
367Disallow: /
368
369User-agent: RepoMonkey
370Disallow: /
371
372User-agent: RMA
373Disallow: /
374
375User-agent: Scrapy
376Disallow: /
377
378User-agent: ShapBot
379Disallow: /
380
381User-agent: SiteBot
382Disallow: /
383
384User-agent: Sogou web spider
385Disallow: /
386
387User-agent: sosospider
388Disallow: /
389
390User-agent: SpankBot
391Disallow: /
392
393User-agent: Speedy
394Disallow: /
395
396User-agent: spotter
397Disallow: /
398
399User-agent: suggybot
400Disallow: /
401
402User-agent: SuperBot
403Disallow: /
404
405User-agent: SuperBot/2.6
406Disallow: /
407
408User-agent: suzuran
409Disallow: /
410
411User-agent: Szukacz/1.4
412Disallow: /
413
414User-agent: Telesoft
415Disallow: /
416
417User-agent: The Intraformant
418Disallow: /
419
420User-agent: TheNomad
421Disallow: /
422
423User-agent: TightTwatBot
424Disallow: /
425
426User-agent: Titan
427Disallow: /
428
429User-agent: toCrawl/UrlDispatcher
430Disallow: /
431
432User-agent: TosCrawler
433Disallow: /
434
435User-agent: True_Robot
436Disallow: /
437
438User-agent: True_Robot/1.0
439Disallow: /
440
441User-agent: turingos
442Disallow: /
443
444User-agent: URLy Warning
445Disallow: /
446
447User-agent: VCI
448Disallow: /
449
450User-agent: vsw
451Disallow: /
452
453User-agent: wapspider
454Disallow: /
455
456User-agent: WebBandit
457Disallow: /
458
459User-agent: WebBandit/3.50
460Disallow: /
461
462User-agent: WebEnhancer
463Disallow: /
464
465User-agent: WebmasterWorldForumBot
466Disallow: /
467
468User-agent: Webster Pro
469Disallow: /
470
471User-agent: WebZinger
472Disallow: /
473
474User-agent: Webzio-Extended
475Disallow: /
476
477User-agent: WWW-Collector-E
478Disallow: /
479
480User-agent: yacy
481Disallow: /
482
483User-agent: yandex
484Disallow: /
485
486User-agent: YRSPider
487Disallow: /
488
489User-agent: Zealbot
490Disallow: /
491
492User-agent: Zeus
493Disallow: /
494
495User-agent: Zite
496Disallow: /
497
498User-agent: Zookabot
499Disallow: /
500
501User-agent: ZyBORG
502Disallow: /
503
504#Outil
505
506User-agent: 5emeRue
507Disallow: /
508
509User-agent: 5erue
510Disallow: /
511
512User-agent: adequat
513Disallow: /
514
515User-agent: adequat-systems
516Disallow: /
517
518User-agent: Amazonbot
519Disallow: /
520
521User-agent: Augure
522Disallow: /
523
524User-agent: auramundi
525Disallow: /
526
527User-agent: BizInformation
528Disallow: /
529
530User-agent: Bloomberg
531Disallow: /
532
533User-agent: CherryPicker
534Disallow: /
535
536User-agent: CherryPickerElite/1.0
537Disallow: /
538
539User-agent: CherryPickerSE/1.0
540Disallow: /
541
542User-agent: Cision
543Disallow: /
544
545User-agent: coexel
546Disallow: /
547
548User-agent: CopyRightCheck
549Disallow: /
550
551User-agent: Corporama
552Disallow: /
553
554User-agent: Digimind
555Disallow: /
556
557User-agent: dotbot
558Disallow: /
559
560User-agent: Download Ninja
561Disallow: /
562
563User-agent: downloadexpress
564Disallow: /
565
566User-agent: ellisphere
567Disallow: /
568
569User-agent: EmailCollector
570Disallow: /
571
572User-agent: EmailSiphon
573Disallow: /
574
575User-agent: EmailWolf
576Disallow: /
577
578User-agent: Europresse
579Disallow: /
580
581User-agent: ExtractorPro
582Disallow: /
583
584User-agent: Fasterfox
585Disallow: /
586
587User-agent: HTTrack
588Disallow: /
589
590User-agent: HTTrack 3.0
591Disallow: /
592
593User-agent: Igentia
594Disallow: /
595
596User-agent: Kantar
597Disallow: /
598
599User-agent: kbcrawl
600Disallow: /
601
602User-agent: Knowings
603Disallow: /
604
605User-agent: leadbox
606Disallow: /
607
608User-agent: LinkextractorPro
609Disallow: /
610
611User-agent: linkfluence
612Disallow: /
613
614User-agent: manageo
615Disallow: /
616
617User-agent: mediacompil
618Disallow: /
619
620User-agent: mention
621Disallow: /
622
623User-agent: Moreover
624Disallow: /
625
626User-agent: mytwip
627Disallow: /
628
629User-agent: NetAnts
630Disallow: /
631
632User-agent: NetMechanic
633Disallow: /
634
635User-agent: newscan-online
636Disallow: /
637
638User-agent: NewsNow
639Disallow: /
640
641User-agent: Offline Explorer
642Disallow: /
643
644User-agent: opinion-tracker
645Disallow: /
646
647User-agent: Qwam content intelligence
648Disallow: /
649
650User-agent: readability.com
651Disallow: /
652
653User-agent: scoop.it
654Disallow: /
655
656User-agent: score3
657Disallow: /
658
659User-agent: SemrushBot
660Disallow: /
661
662User-agent: SightupBot
663Disallow: /
664
665User-agent: Sindup
666Disallow: /
667
668User-agent: SirdataBot
669Disallow: /
670
671User-agent: SiteSnagger
672Disallow: /
673
674User-agent: SiteSucker
675Disallow: /
676
677User-agent: Synthesio
678Disallow: /
679
680User-agent: Talkwalker
681Disallow: /
682
683User-agent: TavilyBot
684Disallow: /
685
686User-agent: Teleport
687Disallow: /
688
689User-agent: TeleportPro
690Disallow: /
691
692User-agent: trendeo
693Disallow: /
694
695User-agent: trendybuzz
696Disallow: /
697
698User-agent: TurnitinBot
699Disallow: /
700
701User-agent: up2news
702Disallow: /
703
704User-agent: UrlPouls
705Disallow: /
706
707User-agent: vecteurplus
708Disallow: /
709
710User-agent: Verif
711Disallow: /
712
713User-agent: verticalsearch
714Disallow: /
715
716User-agent: Web Image Collector
717Disallow: /
718
719User-agent: WebAuto
720Disallow: /
721
722User-agent: WebCopier
723Disallow: /
724
725User-agent: webcopy
726Disallow: /
727
728User-agent: Webedia
729Disallow: /
730
731User-agent: webmirror
732Disallow: /
733
734User-agent: WebReaper
735Disallow: /
736
737User-agent: WebSauger
738Disallow: /
739
740User-agent: website extractor
741Disallow: /
742
743User-agent: Website Quester
744Disallow: /
745
746User-agent: WebStripper
747Disallow: /
748
749User-agent: WebStripper/2.02
750Disallow: /
751
752User-agent: WebZIP
753Disallow: /
754
755User-agent: Wget
756Disallow: /
757
758User-agent: WikioFeedBot
759Disallow: /
760
761User-agent: winello
762Disallow: /
763
764User-agent: WinHTTrack
765Disallow: /
766
767User-agent: Xenu Link Sleuth/1.3.8
768Disallow: /
769
770User-agent: YouBot
771Disallow: /
772
773User-agent: Youmag
774Disallow: /
775Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/articles.xml
776Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/sections.xml
777Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/topics.xml
778Sitemap: https://www.lefigaro.fr/sitemap_news.xml
779Sitemap: https://sitemaps.lefigaro.fr/lefigaro.fr/elections/resultats/sitemap.xml
780Sitemap: https://static.lefigaro.fr/f1/lefigaro/sitemaps/robots-service.txt