🇩🇪 Bayer
bayer.com · European companies · rank #33 · Pharmaceuticals · live robots.txt ↗
AI crawler access (latest snapshot, 13h ago)
⛔blocked
restricted
✅allowed
faded = inherited from the * wildcard group
✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot
Current robots.txt 6757 bytes · sha256 dbce5409b953 · raw
# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/robotstxt.html #User-agent: python-requests/2.20.0 #Disallow: / User-agent: * # CSS, JS, Images Allow: /core/*.css$ Allow: /core/*.css? Allow: /core/*.js$ Allow: /core/*.js? Allow: /core/*.gif Allow: /core/*.jpg Allow: /core/*.jpeg Allow: /core/*.png Allow: /core/*.svg Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /profiles/*.svg # Directories Disallow: /core/ Disallow: /profiles/ Disallow: /sites/default/files/notbeindexed/ # Files Disallow: /README.txt Disallow: /web.config # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/logout/ Disallow: /contact/ Disallow: /kontakt/ Disallow: /report/ Disallow: /event/ Disallow: /product/ Disallow: /uk/ua/v-med-news/overview Disallow: /uk/ua/v-med-news/1 Disallow: /uk/ua/v-med-news/mednews231 Disallow: /uk/ua/v-med-news/anticoagulant-therapy Disallow: /uk/ua/v-med-news/endometriom Disallow: /uk/ua/v-med-news/dienogest Disallow: /uk/ua/v-med-news/contraception Disallow: /uk/ua/v-med-news/2 Disallow: /uk/ua/v-med-news/3 Disallow: /uk/ua/v-med-news/time-interview Disallow: /*/locations$ Disallow: /*cl0n3* # Paths (no clean URLs) Disallow: /index.php/admin/ Disallow: /index.php/comment/reply/ Disallow: /index.php/filter/tips Disallow: /index.php/node/add/ Disallow: /index.php/search/ Disallow: /index.php/user/password/ Disallow: /index.php/user/register/ Disallow: /index.php/user/login/ Disallow: /index.php/user/logout/ Disallow: /*? Disallow: /*?autotranslate=* Disallow: /en/data-privacy-information-for-managing-contingent-workforce Disallow: /*/form/* Disallow: /*/webform/* Disallow: /en/cxpatients Disallow: /en/investors/document-download-tool? Disallow: /de/investoren/document-download-tool? Disallow: /node/* Sitemap: https://www.bayer.com/sitemap.xml # ..::------==------::.. # .::-==========================--:. # .:-=============-----------============--:. # .:-========--::.. ..::---=======-:. # .:-=======-:. -++++++++=-. .::--=====-:. # .-=======-. =##*+++++###: .:--=====-: # .-======-. =##=.....+##= .:--====-: # .-=====-: =###########. .:--===-: # .-=====-. =##- :##* .:--===-. # :=====-: =##*+++++*##- .:--===: # -=====: :=========-. .:--==- # .-=====. ... .::--=-. # .-====-. .####: :::-==. # -====- .##*+##: :::-=-. # -====- .##* +##: :::--- # :====-. .###=---###: :::-=- # .=====: .############: .:::-=. # -====- .###. *##: ::::-- # =====: .::::-. # :===== -++++++++=- -++- .=++. .+++. :++++++++++= .++++++++=: ::::-: # -====- +##++++++###. =####+ .+##= -##*. -##*=======- .##*====+##* .::::- # -====: +##-:::::*##. +##--##= :##*..*##- -##+:::::::. .##+....:### .::::- # -====: +##########* =##- -##= +####*. -##########- .#########*: .::::- # -====: +##: -##= =###****###= =##+ -##= .##+..=##*. .::::: # :-===- +##*+++++###: =##=======+##= -##= -##*+++++++= .##+ .*##- .::::: # .-==== -=========: .==- -==. .==: .==========- ==: -==: :::::. # :-===: .......... .:::::. # ::-==- =##########+ :::::: # :--==: =##-........ .:::::. # ::-==- =##*+++++++: :::::: # ::-==- =##+=======: :::::: # ::--=- =##=........ ::::::. # :::-=- =##########+ ::::::. # :::---. ::::::. # ::::-=: .:::::::::. .:::::: # .:::---. =##########*. ::::::: # ::::---. =##- =##= :::::::. # .:::::--: =###****###*. .:::::::. # .:::::--:. =##=---*##+ .:::::::. # .:::::::-:. =##- =##*. .::::::::. # .:::::::::.. -**: :***: .::::::::::. # .::::::::::::... ..::::::::::::. # .::::::::::::::::::::::::::::::::::::::::. # ..::::::::::::::::::::::::::::::.. # ....::::::::::::::....
Change history
-
initial snapshot
- First snapshot of robots.txt archived