Imperial College London / robots.txt
← back to imperial.ac.uk · change detected 8d ago (2026-10-30T05:39:55Z) · modified +1 −0
What changed
- Added Disallow: /study/courses/postgraduate-taught/2026/ for terminalfour-nutch-spider
Diff old (2026-09-25T17:22:55Z) → new (2026-10-30T05:39:55Z)
| @@ -114,6 +114,7 @@ | ||
| 114 | 114 | Disallow: /study/courses/undergraduate/2025/ |
| 115 | 115 | Disallow: /study/courses/postgraduate-taught/2025/ |
| 116 | 116 | Disallow: /study/courses/undergraduate/2026/ |
| 117 | + Disallow: /study/courses/postgraduate-taught/2026/ | |
| 117 | 118 | |
| 118 | 119 | # old open days |
| 119 | 120 | Disallow: /study/visit/undergraduate/on-demand/ |