Massachusetts Institute of Technology / robots.txt snapshot
← back to mit.edu · fetched 2026-06-26T14:15:22Z (4h ago) · HTTP 200 · 199 bytes · sha256 bad4e8ead1b0414e · raw
final URL: https://web.mit.edu/robots.txt
| 1 | User-agent: * |
| 2 | Disallow: /afs/ |
| 3 | Disallow: /cgi-bin/ |
| 4 | Disallow: /user/ |
| 5 | Disallow: /org/ |
| 6 | Disallow: /activity/ |
| 7 | Disallow: /contrib/ |
| 8 | Disallow: /dept/ |
| 9 | Disallow: /software/ |
| 10 | Disallow: /bin/ |
| 11 | Disallow: */Public/* |