NeuralCrawl

Flickr / robots.txt snapshot

← back to flickr.com · fetched 2026-06-30T06:03:14Z (1mo ago) · HTTP 200 · 2131 bytes · sha256 5ff05803ff23eafe · raw

final URL: https://flickr.com/robots.txt

1User-agent: Adsbot-Google
2User-agent: Alexabot
3User-agent: AppleBot
4User-agent: archive.org_bot
5User-agent: Assaybot
6User-agent: baiduspider
7User-agent: bitlybot
8User-agent: bingbot
9User-agent: BingPreview
10User-agent: ChatGPT-User
11User-agent: Claude-SearchBot
12User-agent: Claude-User
13User-agent: coccocbot
14User-agent: Discordbot
15User-agent: DuckDuckBot
16User-agent: facebookexternalhit
17User-agent: Google-Site-Verification
18User-agent: Google-Sitemaps
19User-agent: Googlebot
20User-agent: Googlebot-Image
21User-agent: Googlebot-Mobile
22User-agent: Googlebot-News
23User-agent: Googlebot-Video
24User-agent: gsa-crawler
25User-Agent: HatenaBlog-bot
26User-agent: ia_archiver
27User-agent: Mediapartners-Google
28User-agent: msnbot
29User-agent: NAVER
30User-agent: OAI-SearchBot
31User-agent: PerplexityBot
32User-agent: PetalBot
33User-agent: Pingdom
34User-agent: Pinterest
35User-agent: redditbot
36User-agent: Slackbot
37User-agent: Slurp
38User-agent: snapchat
39User-agent: Twitterbot
40User-agent: wp.com
41User-agent: Yandex
42User-agent: YandexImages
43User-agent: YandexVideoParser
44User-agent: Yeti
45User-agent: ZoomBot
46Disallow: /gp/
47Disallow: /report_abuse.gne
48Disallow: /abuse
49Disallow: /images/*
50Disallow: /apps/*
51Disallow: /tools/demos/*
52Disallow: /search
53Disallow: /services/oauth
54Disallow: /groups/10millionphotos/
55Disallow: /photos/youpy/
56Disallow: /photos/i_love_u_get_away_from_me/
57Disallow: /faves-i_love_u_get_away_from_me/
58Disallow: /photos/gbachelie/
59Disallow: /photos/archivesact/6011019532/nearby/
60Disallow: /yss_fragment.gne
61Disallow: /photos/tags/*/page*
62Disallow: /photos/*/lightbox/
63Disallow: /groups/*/lightbox/
64Disallow: /explore/*/lightbox/
65
66User-agent: magpie-crawler
67Disallow: /
68
69User-agent: *
70Disallow: /
71
72Sitemap: https://www.flickr.com/sitemap/index/users/sitemap-index-users-00000000.xml.gz
73Sitemap: https://www.flickr.com/sitemap/index/tags/sitemap-index-tags-00000000.xml.gz
74Sitemap: https://www.flickr.com/sitemap/index/sets/sitemap-index-sets-00000000.xml.gz
75Sitemap: https://www.flickr.com/sitemap/index/photos/sitemap-index-photos-00000000.xml.gz
76Sitemap: https://www.flickr.com/sitemap/index/groups/sitemap-index-groups-00000000.xml.gz