Daily Mail / robots.txt
← back to dailymail.co.uk · change detected 1mo ago (2026-09-23T16:51:28Z) · modified +1 −2
What changed
- Removed Allow: /buyline/ for AhrefsBot
- 🤖 AIRemoved Allow: /buyline/ for AI2Bot
- Removed Allow: /buyline/ for Ai2Bot-Dolma
- Removed Allow: /buyline/ for amazon-kendra-
- 🤖 AIRemoved Allow: /buyline/ for Amazonbot
- 🤖 AIRemoved Allow: /buyline/ for anthropic-ai
- 🤖 AIRemoved Allow: /buyline/ for Applebot-Extended
- Removed Allow: /buyline/ for bedrockbot
- 🤖 AIRemoved Allow: /buyline/ for Bytespider
- 🤖 AIRemoved Allow: /buyline/ for CCBot
- Removed Allow: /buyline/ for ChatGLM-Spider
- 🤖 AIRemoved Allow: /buyline/ for ChatGPT-User
- 🤖 AIRemoved Allow: /buyline/ for Claude-SearchBot
- 🤖 AIRemoved Allow: /buyline/ for Claude-User
- 🤖 AIRemoved Allow: /buyline/ for Claude-Web
- 🤖 AIRemoved Allow: /buyline/ for ClaudeBot
- Removed Allow: /buyline/ for CloudflareBrowserRenderingCrawler
- 🤖 AIRemoved Allow: /buyline/ for cohere-ai
- Removed Allow: /buyline/ for Cotoyogi
- Removed Allow: /buyline/ for DeepSeekBot
- 🤖 AIRemoved Allow: /buyline/ for Diffbot
- 🤖 AIRemoved Allow: /buyline/ for DuckAssistBot
- Removed Allow: /buyline/ for EchoboxBot
- 🤖 AIRemoved Allow: /buyline/ for FacebookBot
- Removed Allow: /buyline/ for FriendlyCrawler
- Removed Allow: /buyline/ for Gemini-Deep-Research
- Removed Allow: /buyline/ for Google-CloudVertexBot
- 🤖 AIRemoved Allow: /buyline/ for Google-Extended
- Removed Allow: /buyline/ for GoogleOther-Image
- Removed Allow: /buyline/ for GoogleOther-Video
- 🤖 AIRemoved Allow: /buyline/ for GPTBot
- Removed Allow: /buyline/ for Grok
- Removed Allow: /buyline/ for iaskspider/2.0
- Removed Allow: /buyline/ for ICC-Crawler
- Removed Allow: /buyline/ for ImagesiftBot
- Removed Allow: /buyline/ for img2dataset
- Removed Allow: /buyline/ for ISSCyberRiskCrawler
- Removed Allow: /buyline/ for Kangaroo Bot
- Removed Allow: /buyline/ for KunatoCrawler
- Removed Allow: /buyline/ for LinerBot
- Removed Allow: /buyline/ for Meltwater
- 🤖 AIRemoved Allow: /buyline/ for Meta-ExternalAgent
- 🤖 AIRemoved Allow: /buyline/ for Meta-ExternalFetcher
- 🤖 AIRemoved Allow: /buyline/ for MistralAI-User
- Removed Allow: /buyline/ for OAI-Operator
- 🤖 AIRemoved Allow: /buyline/ for OAI-SearchBot
- 🤖 AIRemoved Allow: /buyline/ for omgili
- Removed Allow: /buyline/ for omgilibot
- 🤖 AIRemoved Allow: /buyline/ for PanguBot
- 🤖 AIRemoved Allow: /buyline/ for Perplexity-User
- ... and 15 more change(s)
Diff old (2026-08-19T17:42:34Z) → new (2026-09-23T16:51:28Z)
| @@ -1,4 +1,4 @@ | ||
| 1 | - # Robots.txt for https://www.dailymail.com/ updated 19/08/2026 | |
| 1 | + # Robots.txt for https://www.dailymail.com/ updated 23/09/2026 | |
| 2 | 2 | # |
| 3 | 3 | # |
| 4 | 4 | # Daily Mail content is made available for your personal and non-commercial |
| @@ -158,7 +158,6 @@ | ||
| 158 | 158 | User-agent: wpbot |
| 159 | 159 | User-agent: WRTNBot |
| 160 | 160 | User-agent: YouBot |
| 161 | - Allow: /buyline/ | |
| 162 | 161 | Allow: /*_.html |
| 163 | 162 | Disallow: / |
| 164 | 163 | # |