NeuralCrawl

Goodreads / robots.txt snapshot

← back to goodreads.com · fetched 2026-06-20T01:10:31Z (18h ago) · HTTP 200 · 2647 bytes · sha256 e1738b2f72796938 · raw

final URL: https://www.goodreads.com/robots.txt

1# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
2
3User-agent: bingbot
4Crawl-delay: 5
5
6User-agent: EtaoSpider
7Disallow: /
8
9User-agent: GPTBot
10Disallow: /
11
12User-agent: CCBot
13Disallow: /
14
15User-agent: AmazonAdBot
16Allow: /book/show
17Allow: /ebooks
18Allow: /photo/group/
19Allow: /search
20Allow: /user/year_in_books
21Allow: /videos
22
23User-agent: facebookexternalhit
24Allow: /user/year_in_books
25
26User-agent: Mediapartners-Google
27Disallow: /about/team_member/
28Disallow: /admin
29Disallow: /api
30Disallow: /blog/list_rss
31Disallow: /book/reviews/
32Disallow: /book_link/follow/
33Disallow: /buy_buttons/
34Disallow: /ebooks
35Disallow: /friend/add_as_friend
36Disallow: /home/index_rss
37Disallow: /oggiPlayerLoader.htm
38Disallow: /photo/group/
39Disallow: /quotes/list_rss
40Disallow: /review/list
41Disallow: /review/list_rss
42Disallow: /review/rate
43Disallow: /shelf/user_shelves
44Disallow: /tooltips
45Disallow: /track
46Disallow: /trivia/answer
47Disallow: /user/updates_rss
48Disallow: /*reviewFilters
49
50User-agent: *
51Allow: /work/editions
52Allow: /work/quotes
53Disallow: /about/team_member/
54Disallow: /admin
55Disallow: /api
56Disallow: /blog/list_rss
57Disallow: /book/reviews/
58Disallow: /book_link/follow/
59Disallow: /buy_buttons/
60Disallow: /ebooks
61Disallow: /friend/add_as_friend
62Disallow: /home/index_rss
63Disallow: /oggiPlayerLoader.htm
64Disallow: /photo/group/
65Disallow: /quotes/list_rss
66Disallow: /review/list
67Disallow: /review/list_rss
68Disallow: /review/rate
69Disallow: /review/show
70Disallow: /search
71Disallow: /shelf/user_shelves
72Disallow: /tooltips
73Disallow: /track
74Disallow: /trivia/answer
75Disallow: /user/updates_rss
76Disallow: /user/year_in_books
77Disallow: /videos/
78Disallow: /work
79Disallow: /*reviewFilters
80
81Sitemap: https://www.goodreads.com/siteindex.author.xml
82Sitemap: https://www.goodreads.com/siteindex.author_community_question.xml
83Sitemap: https://www.goodreads.com/siteindex.award.xml
84Sitemap: https://www.goodreads.com/siteindex.blog.xml
85Sitemap: https://www.goodreads.com/siteindex.book_community_question.xml
86Sitemap: https://www.goodreads.com/siteindex.genre.xml
87Sitemap: https://www.goodreads.com/siteindex.giveaway.xml
88Sitemap: https://www.goodreads.com/siteindex.group.xml
89Sitemap: https://www.goodreads.com/siteindex.index.xml
90Sitemap: https://www.goodreads.com/siteindex.interview.xml
91Sitemap: https://www.goodreads.com/siteindex.list.xml
92Sitemap: https://www.goodreads.com/siteindex.quote.xml
93Sitemap: https://www.goodreads.com/siteindex.quote_tag.xml
94Sitemap: https://www.goodreads.com/siteindex.related_work.xml
95Sitemap: https://www.goodreads.com/siteindex.topic.xml
96Sitemap: https://www.goodreads.com/siteindex.user.xml