NeuralCrawl

TechCrunch / robots.txt snapshot

← back to techcrunch.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 711 bytes · sha256 b6ca480f3ec1899c · raw

final URL: https://techcrunch.com/robots.txt

1User-agent: *
2Disallow: /wp-admin/
3Disallow: /wp-json/
4Allow: /wp-admin/admin-ajax.php
5Disallow: /search/
6Disallow: /?s=
7Disallow: /*?customize_changeset_uuid=*
8
9User-agent: anthropic-ai
10Disallow: /
11
12User-agent: Applebot-Extended
13Disallow: /
14
15User-agent: Bytespider
16Disallow: /
17
18User-agent: CCBot
19Disallow: /
20
21User-agent: ChatGPT-User
22Disallow: /
23
24User-agent: Claude-Web
25User-agent: ClaudeBot
26Disallow: /
27
28User-agent: cohere-ai
29Disallow: /
30
31User-agent: Diffbot
32Disallow: /
33
34User-agent: FacebookBot
35Disallow: /
36
37User-agent: Google-Extended
38Disallow: /
39
40User-agent: GPTBot
41Disallow: /
42
43User-agent: omgili
44Disallow: /
45
46Sitemap: https://techcrunch.com/sitemap.xml
47Sitemap: https://techcrunch.com/news-sitemap.xml
48