NeuralCrawl

HuffPost / robots.txt snapshot

← back to huffpost.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2562 bytes · sha256 a7c45b9f6a0cb53b · raw

final URL: https://www.huffpost.com/robots.txt

1# Cambria robots
2
3User-agent: grapeshot
4Disallow: /member
5Disallow: /*?*err_code=404
6Disallow: /search
7Disallow: /search/?*
8
9User-agent: *
10Disallow: /*?*page=
11Disallow: /member
12Disallow: /*?*err_code=404
13Disallow: /search
14Disallow: /search/?*
15Disallow: /mapi/v4/*/user/*
16Disallow: /embed
17Disallow: /*/webview
18Disallow: /api
19
20User-agent: Amazonbot
21Disallow: /
22
23User-agent: anthropic-ai
24Disallow: /
25
26User-agent: Applebot-Extended
27Disallow: /
28
29User-agent: Bytespider
30Disallow: /
31
32User-agent: CCBot
33Disallow: /
34
35User-agent: ChatGPT-User
36Disallow: /
37
38User-agent: Claude-Web
39Disallow: /
40
41User-agent: ClaudeBot
42Disallow: /
43
44User-agent: cohere-ai
45Disallow: /
46
47User-agent: Diffbot
48Disallow: /
49
50User-agent: DuckAssistBot
51Disallow: /
52
53User-agent: FacebookBot
54Disallow: /
55
56User-agent: GPTBot
57Disallow: /
58
59User-agent: Google-Extended
60Disallow: /
61
62User-agent: Googlebot
63Allow: /
64Disallow: /*?*err_code=404
65Disallow: /search
66Disallow: /search/?*
67
68User-agent: magpie-crawler
69Disallow: /
70
71User-agent: meta-externalagent
72Disallow: /
73
74User-agent: Meta-ExternalFetcher
75Disallow: /
76
77User-agent: OAI-SearchBot
78Disallow: /
79
80User-agent: Perplexity-User
81Disallow: /
82
83User-agent: PerplexityBot
84Disallow: /
85
86User-agent: Timpibot
87Disallow: /
88
89User-agent: TurnitinBot
90Disallow: /
91
92User-agent: Ai2Bot-Dolma
93Disallow: /
94
95User-agent: Claude-SearchBot
96Disallow: /
97
98User-agent: Claude-User
99Disallow: /
100
101User-agent: omgilibot
102Disallow: /
103
104User-agent: TtoYouBot
105Disallow: /
106
107User-agent: WebGPTBot
108Disallow: /
109
110User-agent: Scope3/2.0 (scope3.com)
111Allow: /$
112Allow: /*
113Disallow: /*?*page=
114Disallow: /member
115Disallow: /*?*err_code=404
116Disallow: /search
117Disallow: /search/?*
118Disallow: /mapi/v4/*/user/*
119Disallow: /embed
120Disallow: /api
121
122User-agent: AmazonAdBot
123Allow: /$
124Allow: /*
125Disallow: /*?*page=
126Disallow: /member
127Disallow: /*?*err_code=404
128Disallow: /search
129Disallow: /search/?*
130Disallow: /mapi/v4/*/user/*
131Disallow: /embed
132Disallow: /api
133
134# archives
135Sitemap: https://www.huffpost.com/static-assets/isolated/huffpostsitemapgeneratorjob-prod-public/us/sitemaps/sitemap-v1.xml
136Sitemap: https://www.huffpost.com/static-assets/isolated/huffpostsitemapgeneratorjob-prod-public/us/sitemaps/sitemap-google-news.xml
137Sitemap: https://www.huffpost.com/static-assets/isolated/huffpostsitemapgeneratorjob-prod-public/us/sitemaps/sitemap-google-video.xml
138Sitemap: https://www.huffpost.com/static-assets/isolated/huffpostsitemapgeneratorjob-prod-public/us/sitemaps/sections.xml
139Sitemap: https://www.huffpost.com/static-assets/isolated/huffpostsitemapgeneratorjob-prod-public/us/sitemaps/sitemap-top-sections.xml