NeuralCrawl

๐Ÿ‡ซ๐Ÿ‡ท Safran

safran-group.com · European companies · rank #94 · Aerospace & Defense · live robots.txt ↗

AI crawler access (latest snapshot, 13h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 3133 bytes · sha256 f7a9b71821d8 · raw

#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

User-agent: *
# CSS, JS, Images
Allow: /core/*.css$
Allow: /core/*.css?
Allow: /core/*.js$
Allow: /core/*.js?
Allow: /core/*.gif
Allow: /core/*.jpg
Allow: /core/*.jpeg
Allow: /core/*.png
Allow: /core/*.svg
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /profiles/*.svg
# Directories
Disallow: /core/
Disallow: /profiles/
# Files
Disallow: /README.txt
Disallow: /web.config
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user/logout
# Paths (no clean URLs)
Disallow: /index.php/*
#Disallow: /index.php/admin/
#Disallow: /index.php/comment/reply/
#Disallow: /index.php/filter/tips
#Disallow: /index.php/node/add/
#Disallow: /index.php/search/
#Disallow: /index.php/user/password/
#Disallow: /index.php/user/register/
#Disallow: /index.php/user/login/
#Disallow: /index.php/user/logout/
# Ticket 1170
Disallow: /modal/*
Sitemap: https://www.safran-group.com/sitemap.xml
# ticket 2245
Disallow: /search
Disallow: /fr/recherche
Disallow: /es/buscar
Disallow: /cn/search
# ticket 2281
Disallow: /fr/form*
Disallow: /es/form*
Disallow: /cn/form*
Disallow: /form*
# ticket 2627
Disallow: /*/jobapplication$
Disallow: /*/one-click-jobapplication$
# ticket 2620
user-agent: Googlebot
Disallow: /*?*activities
Disallow: /*?*activity
Disallow: /*?*BoardID
Disallow: /*?*category
Disallow: /*?*companies
Disallow: /*?*company
Disallow: /*?*content
Disallow: /*?*contracts
Disallow: /*?*countries
Disallow: /*?*date_end
Disallow: /*?*date_start
Disallow: /*?*display
Disallow: /*?*duration
Disallow: /*?*experiences
Disallow: /*?*historical_event_type
Disallow: /*?*items_per_page
Disallow: /*?*job_fields
Disallow: /*?*job_status
Disallow: /*?*key_date
Disallow: /*?*lat
Disallow: /*?*location_type
Disallow: /*?*lon
Disallow: /*?*mtm_campaign
Disallow: /*?*mtm_content
Disallow: /*?*mtm_medium
Disallow: /*?*mtm_source
Disallow: /*?*name
Disallow: /*?*OriginID
Disallow: /*?*post_type
Disallow: /*?*qtypes
Disallow: /*?*quarters
Disallow: /*?*radius
Disallow: /*?*regions_states
Disallow: /*?*search
Disallow: /*?*sort_by
Disallow: /*?*sort
Disallow: /*?*tag
Disallow: /*?*topics
Disallow: /*?*trackform
Disallow: /*?*type
Disallow: /*?*utm_content
Disallow: /*?*utm_medium
Disallow: /*?*utm_source
Disallow: /*?*years
Disallow: /*?*product
Disallow: /*?*product_contact
Disallow: /*?*selected_contact
Disallow: /*?*form_type

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived