Mastodon / robots.txt snapshot
← back to mastodon.social · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 231 bytes · sha256 fba77ace66a67c92 · raw
final URL: https://mastodon.social/robots.txt
| 1 | # See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file |
| 2 | |
| 3 | User-agent: GPTBot |
| 4 | Disallow: / |
| 5 | |
| 6 | User-agent: * |
| 7 | Disallow: /media_proxy/ |
| 8 | Disallow: /interact/ |
| 9 | Disallow: /api/v1/instance/domain_blocks |