Daily Mail / robots.txt
← back to dailymail.co.uk · change detected 17d ago (2026-10-20T14:27:19Z) · modified +2 −1
What changed
- Added Allow: /llms.txt for AhrefsBot
- 🤖 AIAdded Allow: /llms.txt for AI2Bot
- Added Allow: /llms.txt for Ai2Bot-Dolma
- Added Allow: /llms.txt for amazon-kendra-
- 🤖 AIAdded Allow: /llms.txt for Amazonbot
- 🤖 AIAdded Allow: /llms.txt for anthropic-ai
- 🤖 AIAdded Allow: /llms.txt for Applebot-Extended
- Added Allow: /llms.txt for bedrockbot
- 🤖 AIAdded Allow: /llms.txt for Bytespider
- 🤖 AIAdded Allow: /llms.txt for CCBot
- Added Allow: /llms.txt for ChatGLM-Spider
- 🤖 AIAdded Allow: /llms.txt for ChatGPT-User
- 🤖 AIAdded Allow: /llms.txt for Claude-SearchBot
- 🤖 AIAdded Allow: /llms.txt for Claude-User
- 🤖 AIAdded Allow: /llms.txt for Claude-Web
- 🤖 AIAdded Allow: /llms.txt for ClaudeBot
- Added Allow: /llms.txt for CloudflareBrowserRenderingCrawler
- 🤖 AIAdded Allow: /llms.txt for cohere-ai
- Added Allow: /llms.txt for Cotoyogi
- Added Allow: /llms.txt for DeepSeekBot
- 🤖 AIAdded Allow: /llms.txt for Diffbot
- 🤖 AIAdded Allow: /llms.txt for DuckAssistBot
- Added Allow: /llms.txt for EchoboxBot
- 🤖 AIAdded Allow: /llms.txt for FacebookBot
- Added Allow: /llms.txt for FriendlyCrawler
- Added Allow: /llms.txt for Gemini-Deep-Research
- Added Allow: /llms.txt for Google-CloudVertexBot
- 🤖 AIAdded Allow: /llms.txt for Google-Extended
- Added Allow: /llms.txt for GoogleOther-Image
- Added Allow: /llms.txt for GoogleOther-Video
- 🤖 AIAdded Allow: /llms.txt for GPTBot
- Added Allow: /llms.txt for Grok
- Added Allow: /llms.txt for iaskspider/2.0
- Added Allow: /llms.txt for ICC-Crawler
- Added Allow: /llms.txt for ImagesiftBot
- Added Allow: /llms.txt for img2dataset
- Added Allow: /llms.txt for ISSCyberRiskCrawler
- Added Allow: /llms.txt for Kangaroo Bot
- Added Allow: /llms.txt for KunatoCrawler
- Added Allow: /llms.txt for LinerBot
- Added Allow: /llms.txt for Meltwater
- 🤖 AIAdded Allow: /llms.txt for Meta-ExternalAgent
- 🤖 AIAdded Allow: /llms.txt for Meta-ExternalFetcher
- 🤖 AIAdded Allow: /llms.txt for MistralAI-User
- Added Allow: /llms.txt for OAI-Operator
- 🤖 AIAdded Allow: /llms.txt for OAI-SearchBot
- 🤖 AIAdded Allow: /llms.txt for omgili
- Added Allow: /llms.txt for omgilibot
- 🤖 AIAdded Allow: /llms.txt for PanguBot
- 🤖 AIAdded Allow: /llms.txt for Perplexity-User
- ... and 15 more change(s)
Diff old (2026-09-23T16:51:28Z) → new (2026-10-20T14:27:19Z)
| @@ -1,4 +1,4 @@ | ||
| 1 | - # Robots.txt for https://www.dailymail.com/ updated 23/09/2026 | |
| 1 | + # Robots.txt for https://www.dailymail.com/ updated 20/10/2026 | |
| 2 | 2 | # |
| 3 | 3 | # |
| 4 | 4 | # Daily Mail content is made available for your personal and non-commercial |
| @@ -159,6 +159,7 @@ | ||
| 159 | 159 | User-agent: WRTNBot |
| 160 | 160 | User-agent: YouBot |
| 161 | 161 | Allow: /*_.html |
| 162 | + Allow: /llms.txt | |
| 162 | 163 | Disallow: / |
| 163 | 164 | # |
| 164 | 165 | #Allow Google sandbox privacy testing |