Daily Mail / robots.txt
← back to dailymail.co.uk · change detected 2d ago (2026-11-05T07:45:14Z) · modified +3 −1
What changed
- Blocked user-agent ‘ExaSearchBot’ entirely
- Blocked user-agent ‘KeenableBot’ entirely
Diff old (2026-10-29T11:22:58Z) → new (2026-11-05T07:45:14Z)
| @@ -1,4 +1,4 @@ | ||
| 1 | - # Robots.txt for https://www.dailymail.com/ updated 29/10/2026 | |
| 1 | + # Robots.txt for https://www.dailymail.com/ updated 04/11/2026 | |
| 2 | 2 | # |
| 3 | 3 | # |
| 4 | 4 | # Daily Mail content is made available for your personal and non-commercial |
| @@ -117,6 +117,7 @@ | ||
| 117 | 117 | User-agent: Diffbot |
| 118 | 118 | User-agent: DuckAssistBot |
| 119 | 119 | User-agent: EchoboxBot |
| 120 | + User-agent: ExaSearchBot | |
| 120 | 121 | User-agent: FacebookBot |
| 121 | 122 | User-agent: FriendlyCrawler |
| 122 | 123 | User-agent: Gemini-Deep-Research |
| @@ -134,6 +135,7 @@ | ||
| 134 | 135 | User-agent: ISSCyberRiskCrawler |
| 135 | 136 | User-agent: Kangaroo Bot |
| 136 | 137 | User-agent: KunatoCrawler |
| 138 | + User-agent: KeenableBot | |
| 137 | 139 | User-agent: LinerBot |
| 138 | 140 | User-agent: Meltwater |
| 139 | 141 | User-agent: Meta-ExternalAgent |