Ireland (Gov.ie) / robots.txt
← back to gov.ie · change detected 1d ago (2026-06-24T10:04:25Z) · modified +53 −2
What changed
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘Applebot’ entirely
- 🤖 AIBlocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘Baiduspider’ entirely
- Blocked user-agent ‘Baiduspider-image’ entirely
- Blocked user-agent ‘Baiduspider-video’ entirely
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- Blocked user-agent ‘DataForSeoBot’ entirely
- Blocked user-agent ‘ERNIEBot’ entirely
- 🤖 AIBlocked Google-Extended (Google) entirely
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- 🤖 AIBlocked meta-externalagent (Meta) entirely
- Blocked user-agent ‘MJ12bot’ entirely
- Blocked user-agent ‘PetalBot’ entirely
- Blocked user-agent ‘Sogou Spider’ entirely
- Blocked user-agent ‘Yandex’ entirely
- Blocked user-agent ‘YiyanBot’ entirely
- Removed Disallow: /admin/ for *
- Removed Disallow: /django-admin/ for *
Diff old (2026-06-20T01:10:30Z) → new (2026-06-24T10:04:25Z)
| @@ -1,6 +1,57 @@ | ||
| 1 | + # Amazon AI | |
| 2 | + User-agent: Amazonbot | |
| 3 | + Disallow: / | |
| 4 | + | |
| 5 | + # Apple Web Crawler | |
| 6 | + User-agent: Applebot | |
| 7 | + User-agent: Applebot-Extended | |
| 8 | + Disallow: / | |
| 9 | + | |
| 10 | + # Baidu | |
| 11 | + User-agent: Baiduspider | |
| 12 | + User-agent: Baiduspider-video | |
| 13 | + User-agent: Baiduspider-image | |
| 14 | + User-agent: ERNIEBot | |
| 15 | + User-agent: YiyanBot | |
| 16 | + Disallow: / | |
| 17 | + | |
| 18 | + # Anthropic AI | |
| 19 | + User-agent: ClaudeBot | |
| 20 | + Disallow: / | |
| 21 | + | |
| 22 | + # DataForSEO | |
| 23 | + User-agent: DataForSeoBot | |
| 24 | + Disallow: / | |
| 25 | + | |
| 26 | + # Google AI | |
| 27 | + User-agent: Google-Extended | |
| 28 | + Disallow: / | |
| 29 | + | |
| 30 | + # OpenAI | |
| 31 | + User-agent: GPTBot | |
| 32 | + Disallow: / | |
| 33 | + | |
| 34 | + # Meta (Facebook, Instagram, WhatsApp) | |
| 35 | + User-agent: Meta-ExternalAgent | |
| 36 | + Disallow: / | |
| 37 | + | |
| 38 | + # Majestic SEO | |
| 39 | + User-agent: MJ12bot | |
| 40 | + Disallow: / | |
| 41 | + | |
| 42 | + # Petal Search | |
| 43 | + User-agent: PetalBot | |
| 44 | + Disallow: / | |
| 45 | + | |
| 46 | + # SoGou | |
| 47 | + User-agent: Sogou Spider | |
| 48 | + Disallow: / | |
| 49 | + | |
| 50 | + # Yandex | |
| 51 | + User-agent: Yandex | |
| 52 | + Disallow: / | |
| 53 | + | |
| 1 | 54 | User-agent: * |
| 2 | - Disallow: /admin/ | |
| 3 | - Disallow: /django-admin/ | |
| 4 | 55 | Disallow: /documents/ |
| 5 | 56 | Disallow: /search/ |
| 6 | 57 | Disallow: /*/search/ |