South China Morning Post / robots.txt
← back to scmp.com · change detected 22h ago (2026-08-05T06:56:07Z) · modified +4 −0
What changed
- Added rules for user-agent ‘facebookexternalhit’
Diff old (2026-07-29T10:28:28Z) → new (2026-08-05T06:56:07Z)
| @@ -23,6 +23,10 @@ | ||
| 23 | 23 | # GrapeShot |
| 24 | 24 | User-agent: grapeshot |
| 25 | 25 | Allow: /*/article/*$ |
| 26 | + | |
| 27 | ||
| 28 | + User-agent: facebookexternalhit | |
| 29 | + Allow: / | |
| 26 | 30 | |
| 27 | 31 | User-agent: * |
| 28 | 32 | Crawl-delay: 10 |