NeuralCrawl

Harvard University / robots.txt snapshot

← back to harvard.edu · fetched 2026-06-26T14:15:22Z (4h ago) · HTTP 200 · 749 bytes · sha256 534d2068c6152177 · raw

final URL: https://www.harvard.edu/robots.txt

1# START YOAST BLOCK
2# ---------------------------
3User-agent: *
4Disallow:
5
6Sitemap: https://www.harvard.edu/sitemap_index.xml
7Sitemap: https://www.harvard.edu/president/sitemap_index.xml
8Sitemap: https://www.harvard.edu/media-relations/sitemap_index.xml
9Sitemap: https://www.harvard.edu/climate-and-sustainability/sitemap_index.xml
10Sitemap: https://www.harvard.edu/wellbeing/sitemap_index.xml
11Sitemap: https://www.harvard.edu/guidelines/sitemap_index.xml
12Sitemap: https://www.harvard.edu/ai/sitemap_index.xml
13Sitemap: https://www.harvard.edu/community/sitemap_index.xml
14Sitemap: https://legacyofslavery.harvard.edu/sitemap_index.xml
15Sitemap: https://www.harvard.edu/federal-lawsuits/sitemap_index.xml
16# ---------------------------
17# END YOAST BLOCK