Stanford University / robots.txt snapshot
← back to stanford.edu · fetched 2026-10-27T16:50:31Z (10d ago) · HTTP 200 · 219 bytes · sha256 ef32c8af1d2e0f97 · raw
final URL: https://www.stanford.edu/robots.txt
| 1 | User-Agent: * |
| 2 | Allow: / |
| 3 | Disallow: /global-components/ |
| 4 | Disallow: /test/ |
| 5 | Disallow: /editor/ |
| 6 | Disallow: /search |
| 7 | Disallow: /storybook/ |
| 8 | Disallow: /examples/ |
| 9 | Disallow: /statuscake |
| 10 | |
| 11 | Sitemap: https://www.stanford.edu/sitemap.xml |