Bayer / robots.txt
← back to bayer.com · change detected 3d ago (2026-11-04T01:20:07Z) · modified +79 −0
What changed
- 🤖 AIBlocked Applebot-Extended (Apple) entirely
- 🤖 AIBlocked Bytespider (ByteDance) entirely
- 🤖 AIBlocked CCBot (Common Crawl) entirely
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- 🤖 AIBlocked Google-Extended (Google) entirely
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- 🤖 AIBlocked meta-externalagent (Meta) entirely
- Added rules for user-agent ‘MistralAI-Index’
- 🤖 AIAdded rules for AI crawler MistralAI-User (Mistral AI)
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
Diff old (2026-10-12T17:28:07Z) → new (2026-11-04T01:20:07Z)
| @@ -1,7 +1,86 @@ | ||
| 1 | 1 | # Bayer Global robots.txt |
| 2 | 2 | # https://www.bayer.com/ |
| 3 | 3 | |
| 4 | + # Default rules for all compliant crawlers | |
| 4 | 5 | User-agent: * |
| 5 | 6 | Disallow: /search/ |
| 6 | 7 | Disallow: /*? |
| 8 | + | |
| 9 | + # OpenAI: allow ChatGPT Search | |
| 10 | + User-agent: OAI-SearchBot | |
| 11 | + Allow: / | |
| 12 | + Disallow: /search/ | |
| 13 | + Disallow: /*? | |
| 14 | + | |
| 15 | + # OpenAI: allow user-requested retrieval | |
| 16 | + User-agent: ChatGPT-User | |
| 17 | + Allow: / | |
| 18 | + Disallow: /search/ | |
| 19 | + Disallow: /*? | |
| 20 | + | |
| 21 | + # OpenAI: block model-training crawler | |
| 22 | + User-agent: GPTBot | |
| 23 | + Disallow: / | |
| 24 | + | |
| 25 | + # Anthropic: allow Claude Search | |
| 26 | + User-agent: Claude-SearchBot | |
| 27 | + Allow: / | |
| 28 | + Disallow: /search/ | |
| 29 | + Disallow: /*? | |
| 30 | + | |
| 31 | + # Anthropic: allow user-requested retrieval | |
| 32 | + User-agent: Claude-User | |
| 33 | + Allow: / | |
| 34 | + Disallow: /search/ | |
| 35 | + Disallow: /*? | |
| 36 | + | |
| 37 | + # Anthropic: block model-training crawler | |
| 38 | + User-agent: ClaudeBot | |
| 39 | + Disallow: / | |
| 40 | + | |
| 41 | + # Perplexity: allow search indexing and citations | |
| 42 | + User-agent: PerplexityBot | |
| 43 | + Allow: / | |
| 44 | + Disallow: /search/ | |
| 45 | + Disallow: /*? | |
| 46 | + | |
| 47 | + # Perplexity: allow user-requested retrieval | |
| 48 | + User-agent: Perplexity-User | |
| 49 | + Allow: / | |
| 50 | + Disallow: /search/ | |
| 51 | + Disallow: /*? | |
| 52 | + | |
| 53 | + # Mistral: allow AI search indexing | |
| 54 | + User-agent: MistralAI-Index | |
| 55 | + Allow: / | |
| 56 | + Disallow: /search/ | |
| 57 | + Disallow: /*? | |
| 58 | + | |
| 59 | + # Mistral: allow user-requested retrieval | |
| 60 | + User-agent: MistralAI-User | |
| 61 | + Allow: / | |
| 62 | + Disallow: /search/ | |
| 63 | + Disallow: /*? | |
| 64 | + | |
| 65 | + # Google: opt out of Gemini model training and grounding | |
| 66 | + # Does not affect Google Search, AI Overviews or AI Mode | |
| 67 | + User-agent: Google-Extended | |
| 68 | + Disallow: / | |
| 69 | + | |
| 70 | + # Apple: opt out of Apple generative-AI model training | |
| 71 | + # Does not block regular Applebot search crawling | |
| 72 | + User-agent: Applebot-Extended | |
| 73 | + Disallow: / | |
| 74 | + | |
| 75 | + # Meta: block model-training crawler | |
| 76 | + User-agent: meta-externalagent | |
| 77 | + Disallow: / | |
| 78 | + | |
| 79 | + # Common Crawl: block collection for public datasets | |
| 80 | + User-agent: CCBot | |
| 81 | + Disallow: / | |
| 82 | + | |
| 83 | + # ByteDance: block AI-training crawler | |
| 84 | + User-agent: Bytespider | |
| 85 | + Disallow: / | |
| 7 | 86 | Sitemap: https://www.bayer.com/sitemap.xml |