NeuralCrawl

TotalEnergies / robots.txt snapshot

← back to totalenergies.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 4248 bytes · sha256 3e614c1c032f938d · raw

final URL: https://totalenergies.com/robots.txt

1#
2# robots.txt
3#
4# This file is to prevent the crawling and indexing of certain parts
5# of your site by web crawlers and spiders run by sites like Yahoo!
6# and Google. By telling these "robots" where not to go on your site,
7# you save bandwidth and server resources.
8#
9# This file will be ignored unless it is at the root of your host:
10# Used: http://example.com/robots.txt
11# Ignored: http://example.com/site/robots.txt
12#
13# For more information about the robots.txt standard, see:
14# http://www.robotstxt.org/robotstxt.html
15
16User-agent: *
17# CSS, JS, Images
18Allow: /core/*.css$
19Allow: /core/*.css?
20Allow: /core/*.js$
21Allow: /core/*.js?
22Allow: /core/*.gif
23Allow: /core/*.jpg
24Allow: /core/*.jpeg
25Allow: /core/*.png
26Allow: /core/*.svg
27Allow: /profiles/*.css$
28Allow: /profiles/*.css?
29Allow: /profiles/*.js$
30Allow: /profiles/*.js?
31Allow: /profiles/*.gif
32Allow: /profiles/*.jpg
33Allow: /profiles/*.jpeg
34Allow: /profiles/*.png
35Allow: /profiles/*.svg
36# Directories
37Disallow: /core/
38Disallow: /profiles/
39# Files
40Disallow: /README.txt
41Disallow: /web.config
42# Paths (clean URLs)
43Disallow: /comment/reply/
44Disallow: /filter/tips
45Disallow: /node/add/
46Allow: /search/
47Allow: /recherche/
48# Paths (no clean URLs)
49Disallow: /index.php/comment/reply/
50Disallow: /index.php/filter/tips
51Disallow: /index.php/node/add/
52Disallow: /index.php/search/
53Disallow: /index.php/user/logout/
54# Search (with parameters)
55Allow: /search/content?*
56Allow: /recherche/contenu?*
57Allow: */medias/actualites*page=*
58Allow: */media/news*page=*
59Allow: */medias/medias*page=*
60Allow: */media/media*page=*
61Disallow: */formulaire-de-contact/*
62Disallow: */contact-form/*
63Disallow: /search-content*
64Disallow: */recherche-contenu*
65# Integration (static files)
66Disallow: /themes/custom/*/integration/
67Disallow: */contenu/publications/*
68
69# -------------------------  
70# Newsroom  
71# -------------------------  
72# 1) Newsroom home (FR/EN) – allow only lang=fra or lang=eng  
73Allow: */newsroom/fr/?lang=fra  
74Allow: */newsroom/en/?lang=eng  
75Disallow: */newsroom/fr/?  
76Disallow: */newsroom/en/?  
77  
78# 2) Communiqués de presse – allow only all-themes (+ pagination)  
79Allow: */newsroom/section/communiques-de-presse/?lang=fra&topic=all-themes$  
80Allow: */newsroom/section/communiques-de-presse/?lang=eng&topic=all-themes$  
81Allow: */newsroom/section/communiques-de-presse/?lang=fra&topic=all-themes&pages=  
82Allow: */newsroom/section/communiques-de-presse/?lang=eng&topic=all-themes&pages=  
83Disallow: */newsroom/section/communiques-de-presse/? 
84  
85# 3) Dossiers de presse – allow only lang param (FR/EN)  
86Allow: */newsroom/section/dossiers-de-presse/?lang=fra  
87Allow: */newsroom/section/dossiers-de-presse/?lang=eng  
88Disallow: */newsroom/section/dossiers-de-presse/?  
89  
90# 4) Revues de presse – allow only vu-dans-la-presse (+ pagination) FR/EN  
91Allow: */newsroom/section/revues-de-presse/?lang=fra&cat=vu-dans-la-presse$  
92Allow: */newsroom/section/revues-de-presse/?lang=eng&cat=vu-dans-la-presse$  
93Allow: */newsroom/section/revues-de-presse/?lang=fra&cat=vu-dans-la-presse&pages=  
94Allow: */newsroom/section/revues-de-presse/?lang=eng&cat=vu-dans-la-presse&pages=  
95Disallow: */newsroom/section/revues-de-presse/?  
96  
97# 5) Contacts – allow only lang param (FR/EN)  
98Allow: */newsroom/section/contacts/?lang=fra  
99Allow: */newsroom/section/contacts/?lang=eng  
100Disallow: */newsroom/section/contacts/?  
101  
102# 6) Search – disallow all  
103Disallow: */newsroom/?s  
104Disallow: */newsroom/?lang=eng&s  
105Disallow: */newsroom/?lang=fra&s  
106  
107# 7) Statements – allow lang only & all-themes (+ pagination)   
108Allow: */newsroom/section/statements/?lang=fra$   
109Allow: */newsroom/section/statements/?lang=eng$   
110Allow: */newsroom/section/statements/?lang=fra&topic=all-themes$   
111Allow: */newsroom/section/statements/?lang=eng&topic=all-themes$   
112Allow: */newsroom/section/statements/?lang=fra&pages=   
113Allow: */newsroom/section/statements/?lang=eng&pages=   
114Allow: */newsroom/section/statements/?lang=fra&topic=all-themes&pages=  
115Allow: */newsroom/section/statements/?lang=eng&topic=all-themes&pages=  
116Disallow: */newsroom/section/statements/? 
117 
118# 8) Selection – disallow all 
119Disallow: */newsroom/fr/selection/ 
120Disallow: */newsroom/en/selection/ 
121Sitemap: https://totalenergies.com/sitemap.xml