NeuralCrawl

Air Liquide / robots.txt snapshot

← back to airliquide.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 3131 bytes · sha256 610d38f12a999f3f · raw

final URL: https://www.airliquide.com/robots.txt

1#
2# robots.txt
3#
4# This file is to prevent the crawling and indexing of certain parts
5# of your site by web crawlers and spiders run by sites like Yahoo!
6# and Google. By telling these "robots" where not to go on your site,
7# you save bandwidth and server resources.
8#
9# This file will be ignored unless it is at the root of your host:
10# Used: http://example.com/robots.txt
11# Ignored: http://example.com/site/robots.txt
12#
13# For more information about the robots.txt standard, see:
14# http://www.robotstxt.org/robotstxt.html
15
16User-agent: *
17# CSS, JS, Images
18Allow: /core/*.css$
19Allow: /core/*.css?
20Allow: /core/*.js$
21Allow: /core/*.js?
22Allow: /core/*.gif
23Allow: /core/*.jpg
24Allow: /core/*.jpeg
25Allow: /core/*.png
26Allow: /core/*.svg
27Allow: /profiles/*.css$
28Allow: /profiles/*.css?
29Allow: /profiles/*.js$
30Allow: /profiles/*.js?
31Allow: /profiles/*.gif
32Allow: /profiles/*.jpg
33Allow: /profiles/*.jpeg
34Allow: /profiles/*.png
35Allow: /profiles/*.svg
36Allow: /sites/*/files/
37# Directories
38Disallow: /core/
39Disallow: /profiles/
40# Files
41Disallow: /README.txt
42Disallow: /web.config
43# Paths (clean URLs)
44Disallow: /admin/
45Disallow: /comment/reply/
46Disallow: /filter/tips
47Disallow: /node/add/
48Disallow: /search/
49Disallow: /user/register/
50Disallow: /user/password/
51Disallow: /user/login/
52Disallow: /user/logout/
53# Paths (no clean URLs)
54Disallow: /index.php/admin/
55Disallow: /index.php/comment/reply/
56Disallow: /index.php/filter/tips
57Disallow: /index.php/node/add/
58Disallow: /index.php/search/
59Disallow: /index.php/user/password/
60Disallow: /index.php/user/register/
61Disallow: /index.php/user/login/
62Disallow: /index.php/user/logout/
63# Specific rules
64Disallow: */saml_login
65Disallow: */add-to-calendar/ics/*
66Disallow: */api/airliquide/download/file/*
67Disallow: */form/*
68Disallow: */opinion_survey_json/add/*
69Disallow: */aggregate
70Disallow: */page_action/*
71Disallow: */spa/*
72Disallow: */ajax/*
73Disallow: */jserrors/*
74Disallow: */metrics/*
75Disallow: */page_view_timing/*
76Disallow: */page_view_event/*
77Disallow: */session_trace/*
78# Blocking all parameters
79Disallow: *=*
80Disallow: */node*
81# Except those
82Allow: *page=*
83Allow: *languageSelect=*
84Allow: *thematic%5B0%5D=*
85Allow: *field_date_range_end_value=&field_date_range_end_value_1=&page=*
86Allow: *period%5Bmin%5D=&period%5Bmax%5D=&text=&page=*
87Allow: *period%5Bmin%5D=&period%5Bmax%5D=&page=*
88Allow: *.gif*
89Allow: *.jpg*
90Allow: *.jpeg*
91Allow: *.png*
92Allow: *.webp*
93# Blocking some PDFs
94Disallow: /sites/airliquide.com/files/2023-03/air-liquide-rapport-de-developpement-durable-2022.pdf
95Disallow: /sites/airliquide.com/files/2023-03/air-liquide-sustainability-report-2022.pdf
96Disallow: /sites/airliquide.com/files/2022-04/2021-sustainability-report.pdf
97Disallow: /sites/airliquide.com/files/2022-04/rapport-developpement-durable-2021.pdf
98Disallow: /group/press-releases-news/2023-03-24/sustainability-report-2022-air-liquide-presents-its-results-and-sets-additional-objectives
99Disallow: /fr/groupe/communiques-presse-actualites/24-03-2023/rapport-de-developpement-durable-2022-air-liquide-presente-ses-resultats-et-se-fixe-des-objectifs
100# XML sitemap
101Sitemap: https://www.airliquide.com/sitemap.xml