EPFL / robots.txt
← back to epfl.ch · change detected 21d ago (2026-10-16T12:54:52Z) · modified +0 −130
What changed
- Removed all rules for user-agent ‘*’
- 🤖 AIUnblocked Amazonbot (Amazon) - rules removed
- 🤖 AIUnblocked anthropic-ai (Anthropic) - rules removed
- 🤖 AIUnblocked Applebot-Extended (Apple) - rules removed
- Removed all rules for user-agent ‘atlassian-bot’
- Removed all rules for user-agent ‘AwarioRssBot’
- Removed all rules for user-agent ‘AwarioSmartBot’
- Removed all rules for user-agent ‘Baiduspider’
- Removed all rules for user-agent ‘BorderxBot’
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- 🤖 AIUnblocked CCBot (Common Crawl) - rules removed
- Removed all rules for user-agent ‘CitibotSiteCrawler’
- 🤖 AIUnblocked Claude-Web (Anthropic) - rules removed
- 🤖 AIUnblocked ClaudeBot (Anthropic) - rules removed
- 🤖 AIUnblocked cohere-ai (Cohere) - rules removed
- Removed all rules for user-agent ‘Cotoyogi’
- 🤖 AIUnblocked Diffbot (Diffbot) - rules removed
- Removed all rules for user-agent ‘FishBot’
- Removed all rules for user-agent ‘Google-CloudVertexBot’
- 🤖 AIUnblocked Google-Extended (Google) - rules removed
- Removed all rules for user-agent ‘GoogleOther’
- 🤖 AIUnblocked GPTBot (OpenAI) - rules removed
- Removed all rules for user-agent ‘ICC-Crawler’
- Removed all rules for user-agent ‘KimiBot’
- Removed all rules for user-agent ‘magpie-crawler’
- 🤖 AIUnblocked meta-externalagent (Meta) - rules removed
- Removed all rules for user-agent ‘MistralAI-Training’
- Removed all rules for user-agent ‘NavuBot’
- 🤖 AIUnblocked omgili (Webz.io) - rules removed
- Removed all rules for user-agent ‘PetalBot’
- Removed all rules for user-agent ‘QualifiedBot’
- Removed all rules for user-agent ‘SemrushBot-SWA’
- Removed all rules for user-agent ‘WARDBot’
Diff old (2026-10-12T11:21:56Z) → new (2026-10-16T12:54:52Z)
| @@ -1,130 +1,0 @@ | ||
| 1 | - # As a condition of accessing this website, you agree to abide by the following | |
| 2 | - # content signals: | |
| 3 | - | |
| 4 | - # (a) If a Content-Signal = yes, you may collect content for the corresponding | |
| 5 | - # use. | |
| 6 | - # (b) If a Content-Signal = no, you may not collect content for the | |
| 7 | - # corresponding use. | |
| 8 | - # (c) If the website operator does not include a Content-Signal for a | |
| 9 | - # corresponding use, the website operator neither grants nor restricts | |
| 10 | - # permission via Content-Signal with respect to the corresponding use. | |
| 11 | - | |
| 12 | - # The content signals and their meanings are: | |
| 13 | - | |
| 14 | - # search: building a search index and providing search results (e.g., returning | |
| 15 | - # hyperlinks and short excerpts from your website's contents). Search does not | |
| 16 | - # include providing AI-generated search summaries. | |
| 17 | - # ai-input: inputting content into one or more AI models (e.g., retrieval | |
| 18 | - # augmented generation, grounding, or other real-time taking of content for | |
| 19 | - # generative AI search answers). | |
| 20 | - # ai-train: training or fine-tuning AI models. | |
| 21 | - # use: how AI systems may consume the content (immediate, reference, or full). | |
| 22 | - | |
| 23 | - # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF | |
| 24 | - # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT | |
| 25 | - # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. | |
| 26 | - | |
| 27 | - # BEGIN Cloudflare Managed content | |
| 28 | - | |
| 29 | - | |
| 30 | - User-agent: * | |
| 31 | - Content-Signal: search=yes,ai-train=no,use=reference | |
| 32 | - Allow: / | |
| 33 | - | |
| 34 | - # Training crawlers | |
| 35 | - User-agent: Amazonbot | |
| 36 | - Disallow: / | |
| 37 | - | |
| 38 | - User-agent: Applebot-Extended | |
| 39 | - Disallow: / | |
| 40 | - | |
| 41 | - User-agent: Bytespider | |
| 42 | - Disallow: / | |
| 43 | - | |
| 44 | - User-agent: CCBot | |
| 45 | - Disallow: / | |
| 46 | - | |
| 47 | - User-agent: ClaudeBot | |
| 48 | - Disallow: / | |
| 49 | - | |
| 50 | - User-agent: Diffbot | |
| 51 | - Disallow: / | |
| 52 | - | |
| 53 | - User-agent: Google-Extended | |
| 54 | - Disallow: / | |
| 55 | - | |
| 56 | - User-agent: GPTBot | |
| 57 | - Disallow: / | |
| 58 | - | |
| 59 | - User-agent: omgili | |
| 60 | - Disallow: / | |
| 61 | - | |
| 62 | - User-agent: anthropic-ai | |
| 63 | - Disallow: / | |
| 64 | - | |
| 65 | - User-agent: Claude-Web | |
| 66 | - Disallow: / | |
| 67 | - | |
| 68 | - User-agent: cohere-ai | |
| 69 | - Disallow: / | |
| 70 | - | |
| 71 | - User-agent: MistralAI-Training | |
| 72 | - Disallow: / | |
| 73 | - | |
| 74 | - User-agent: meta-externalagent | |
| 75 | - Disallow: / | |
| 76 | - | |
| 77 | - User-agent: GoogleOther | |
| 78 | - Disallow: / | |
| 79 | - | |
| 80 | - User-agent: Baiduspider | |
| 81 | - Disallow: / | |
| 82 | - | |
| 83 | - User-agent: PetalBot | |
| 84 | - Disallow: / | |
| 85 | - | |
| 86 | - User-agent: AwarioSmartBot | |
| 87 | - User-agent: AwarioRssBot | |
| 88 | - Disallow: / | |
| 89 | - | |
| 90 | - User-agent: Google-CloudVertexBot | |
| 91 | - Disallow: / | |
| 92 | - | |
| 93 | - User-agent: QualifiedBot | |
| 94 | - Disallow: / | |
| 95 | - | |
| 96 | - User-agent: Cotoyogi | |
| 97 | - Disallow: / | |
| 98 | - | |
| 99 | - User-agent: ICC-Crawler | |
| 100 | - Disallow: / | |
| 101 | - | |
| 102 | - User-agent: atlassian-bot | |
| 103 | - Disallow: / | |
| 104 | - | |
| 105 | - User-agent: FishBot | |
| 106 | - Disallow: / | |
| 107 | - | |
| 108 | - User-agent: BorderxBot | |
| 109 | - Disallow: / | |
| 110 | - | |
| 111 | - User-agent: NavuBot | |
| 112 | - Disallow: / | |
| 113 | - | |
| 114 | - User-agent: SemrushBot-SWA | |
| 115 | - Disallow: / | |
| 116 | - | |
| 117 | - User-agent: WARDBot | |
| 118 | - Disallow: / | |
| 119 | - | |
| 120 | - User-agent: magpie-crawler | |
| 121 | - Disallow: / | |
| 122 | - | |
| 123 | - User-agent: KimiBot | |
| 124 | - Disallow: / | |
| 125 | - | |
| 126 | - User-agent: CitibotSiteCrawler | |
| 127 | - Disallow: / | |
| 128 | - | |
| 129 | - # END Cloudflare Managed Content | |
| 130 | - |