NeuralCrawl

Behance / robots.txt snapshot

← back to behance.net · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 1317 bytes · sha256 1a55abe2475b1430 · raw

final URL: https://www.behance.net/robots.txt

1Sitemap: https://www.behance.net/sitemap
2
3# Disallow AI bots
4User-agent: AI2Bot
5User-agent: anthropic-ai
6User-agent: Applebot-Extended
7User-agent: Bytespider
8User-agent: CCbot
9User-agent: Claude-Web
10User-agent: ClaudeBot
11User-agent: cohere-training-data-crawler
12User-agent: Diffbot
13User-agent: FacebookBot
14User-agent: Google-Extended
15User-agent: GPTBot
16User-agent: ImagesiftBot
17User-agent: Kangaroo Bot
18User-agent: Meta-ExternalAgent
19User-agent: Omgili
20User-agent: Omgilibot
21User-agent: PanguBot
22User-agent: Timpibot
23User-agent: Webzio-Extended
24Disallow: /
25
26User-agent: *
27Disallow: /v2/
28Disallow: /v3/
29Disallow: /log$
30Disallow: /search/*?*assetsCategory=*
31Disallow: /search/*?*assetsPriceTier=*
32Disallow: /search/*?*assetsExtensionTypes=*
33Disallow: /search/*?*color_hex=*
34Disallow: /search/*?*country=*
35Disallow: /search/*?*exif_aperture=*
36Disallow: /search/*?*exif_focal_length=*
37Disallow: /search/*?*exif_lens_brand=*
38Disallow: /search/*?*exif_make=*
39Disallow: /search/*?*exif_shutter_speed=*
40Disallow: /search/*?*exif_zoom=*
41Disallow: /search/*?*field=*
42Disallow: /search/*?*gif=*
43Disallow: /search/*?*premium=*
44Disallow: /search/*?*sort=*
45Disallow: /search/*?*schools=*
46Disallow: /search/*?*tools=*
47Disallow: /search/*?*time=*
48Disallow: /search/*?*userAvailability=*
49Disallow: /hire/browse
50Disallow: /*/undefined