Daily Mail / robots.txt
← back to dailymail.co.uk · change detected 8d ago (2026-10-29T11:22:58Z) · modified +5 −1
What changed
- Blocked user-agent ‘cohere-training-data-crawler’ entirely
- Blocked user-agent ‘GPTBot-Search’ entirely
- 🤖 AIBlocked Meta-WebIndexer (Meta) entirely
- Blocked user-agent ‘xAI-SearchBot’ entirely
Diff old (2026-10-20T14:27:19Z) → new (2026-10-29T11:22:58Z)
| @@ -1,4 +1,4 @@ | ||
| 1 | - # Robots.txt for https://www.dailymail.com/ updated 20/10/2026 | |
| 1 | + # Robots.txt for https://www.dailymail.com/ updated 29/10/2026 | |
| 2 | 2 | # |
| 3 | 3 | # |
| 4 | 4 | # Daily Mail content is made available for your personal and non-commercial |
| @@ -111,6 +111,7 @@ | ||
| 111 | 111 | User-agent: Claude-User |
| 112 | 112 | User-agent: Claude-Web |
| 113 | 113 | User-agent: cohere-ai |
| 114 | + User-agent: cohere-training-data-crawler | |
| 114 | 115 | User-agent: Cotoyogi |
| 115 | 116 | User-agent: DeepSeekBot |
| 116 | 117 | User-agent: Diffbot |
| @@ -124,6 +125,7 @@ | ||
| 124 | 125 | User-agent: GoogleOther-Image |
| 125 | 126 | User-agent: GoogleOther-Video |
| 126 | 127 | User-agent: GPTBot |
| 128 | + User-agent: GPTBot-Search | |
| 127 | 129 | User-agent: Grok |
| 128 | 130 | User-agent: iaskspider/2.0 |
| 129 | 131 | User-agent: ICC-Crawler |
| @@ -136,6 +138,7 @@ | ||
| 136 | 138 | User-agent: Meltwater |
| 137 | 139 | User-agent: Meta-ExternalAgent |
| 138 | 140 | User-agent: Meta-ExternalFetcher |
| 141 | + User-agent: Meta-WebIndexer | |
| 139 | 142 | User-agent: MistralAI-User |
| 140 | 143 | User-agent: OAI-Operator |
| 141 | 144 | User-agent: OAI-SearchBot |
| @@ -157,6 +160,7 @@ | ||
| 157 | 160 | User-agent: Webzio-Extended |
| 158 | 161 | User-agent: wpbot |
| 159 | 162 | User-agent: WRTNBot |
| 163 | + User-agent: xAI-SearchBot | |
| 160 | 164 | User-agent: YouBot |
| 161 | 165 | Allow: /*_.html |
| 162 | 166 | Allow: /llms.txt |