California Institute of Technology / robots.txt snapshot
← back to caltech.edu · fetched 2026-06-26T14:15:22Z (4h ago) · HTTP 200 · 266 bytes · sha256 3f591e8c02a0b220 · raw
final URL: https://www.caltech.edu/robots.txt
| 1 | User-agent: SemrushBot |
| 2 | Disallow: / |
| 3 | |
| 4 | User-agent: BLP_bbot |
| 5 | Disallow: / |
| 6 | |
| 7 | User-agent: * |
| 8 | Disallow: /campus-life-events/calendar/minicalendar/* |
| 9 | Disallow: /map/landmark_ajax/* |
| 10 | Disallow: /map/milestone/* |
| 11 | Crawl-delay: 10 |
| 12 | Allow: * |
| 13 | Sitemap: https://www.caltech.edu/sitemap.xml |