The Telegraph / robots.txt
← back to telegraph.co.uk · change detected 30d ago (2026-10-08T10:00:34Z) · modified +4 −1
What changed
- Formatting/comment-only change (no semantic difference)
Diff old (2026-10-05T15:06:47Z) → new (2026-10-08T10:00:34Z)
| @@ -107,11 +107,12 @@ | ||
| 107 | 107 | User-agent: MistralAI-User |
| 108 | 108 | User-agent: OAI-Operator |
| 109 | 109 | User-agent: Perplexity-User |
| 110 | + Disallow: / | |
| 110 | 111 | Allow: /travel/explore-dubai/ |
| 111 | 112 | Allow: /travel/luxury-stays/ |
| 112 | 113 | Allow: /business/uk-business-growth/ |
| 113 | 114 | Allow: /ads.txt |
| 114 | - Disallow: / | |
| 115 | + | |
| 115 | 116 | |
| 116 | 117 | # Sitemap URLs |
| 117 | 118 | Sitemap: https://www.telegraph.co.uk/sitemap.xml |
| @@ -123,3 +124,5 @@ | ||
| 123 | 124 | |
| 124 | 125 | |
| 125 | 126 | |
| 127 | + | |
| 128 | + |