Archive of Our Own / robots.txt snapshot
← back to archiveofourown.org · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 872 bytes · sha256 1493972bb388474f · raw
final URL: https://archiveofourown.org/robots.txt
| 1 | # See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file |
| 2 | |
| 3 | User-agent: * |
| 4 | Disallow: /works? # cruel but efficient |
| 5 | Disallow: /autocomplete/ |
| 6 | Disallow: /downloads/ |
| 7 | Disallow: /external_works/ |
| 8 | # disallow indexing of search results |
| 9 | Disallow: /bookmarks/search? |
| 10 | Disallow: /people/search? |
| 11 | Disallow: /tags/search? |
| 12 | Disallow: /works/search? |
| 13 | |
| 14 | User-agent: Googlebot |
| 15 | Disallow: /autocomplete/ |
| 16 | Disallow: /downloads/ |
| 17 | Disallow: /external_works/ |
| 18 | # Googlebot is smart and knows pattern matching |
| 19 | Disallow: /works/*? |
| 20 | Disallow: /*search? |
| 21 | Disallow: /*?*query= |
| 22 | Disallow: /*?*sort_ |
| 23 | Disallow: /*?*selected_tags |
| 24 | Disallow: /*?*view_adult |
| 25 | Disallow: /*?*tag_id |
| 26 | Disallow: /*?*pseud_id |
| 27 | Disallow: /*?*user_id |
| 28 | Disallow: /*?*pseud= |
| 29 | |
| 30 | User-agent: CCBot |
| 31 | Disallow: / |
| 32 | |
| 33 | User-agent: GPTBot |
| 34 | Disallow: / |
| 35 | |
| 36 | User-agent: ChatGPT-User |
| 37 | Disallow: / |
| 38 | |
| 39 | User-agent: Slurp |
| 40 | Crawl-delay: 30 |