Open Science Framework / robots.txt snapshot
← back to osf.io · fetched 2026-06-26T14:15:22Z (4h ago) · HTTP 200 · 255 bytes · sha256 55143ee8903b3975 · raw
final URL: https://osf.io/robots.txt
| 1 | # www.robotstxt.org/ |
| 2 | |
| 3 | User-agent: * |
| 4 | Disallow: /api/* |
| 5 | Disallow: *?view_only= |
| 6 | crawl-delay: 10 |
| 7 | |
| 8 | # Robots that have misbehaved |
| 9 | User-agent: PingBot |
| 10 | User-agent: PerplexityBot |
| 11 | User-agent: GPTBot |
| 12 | User-agent: BaiduSpider |
| 13 | User-agent: Meta-ExternalAgent |
| 14 | Disallow: * |