NeuralCrawl

Politico / robots.txt snapshot

← back to politico.com · fetched 2026-06-20T11:49:21Z (7h ago) · HTTP 200 · 4674 bytes · sha256 4d85a9821882bcc4 · raw

final URL: manual:file

1Sitemap: https://www.politico.com/sitemap.xml
2Sitemap: https://www.politico.com/news-sitemap.xml
3Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml
4Sitemap: https://www.politico.com/liveupdates-sitemap-latest.xml
5Sitemap: https://www.politico.com/liveupdates-sitemap-content.xml
6
7User-agent: *
8Disallow: /2step
9Disallow: /campaigntrailnotused
10Disallow: /CandidateRanking
11Disallow: /candidates
12Disallow: /CustomCompanionTest
13Disallow: /debate
14Disallow: /ryan
15Disallow: /promotions
16Disallow: /2014-election/results/mobile/iphone
17Disallow: /2014-election/results/mobile/ipad
18Disallow: /_preview
19Disallow: /326home
20Disallow: /327home
21Disallow: /_styleguide
22Disallow: /newsletters/politico-toplines
23Disallow : /*?tab=most-read*
24Disallow : /*?ml=*
25Disallow : /*?hp=*
26Disallow : /*?video=*
27Disallow : /*?nid=*
28Disallow : /*?cid=*
29Disallow : /*?source=*
30Disallow : /*?slide=*
31Disallow : /*?category=*
32Disallow : /*?p=*
33Disallow : /*?print=*
34
35
36User-agent: Googlebot-News
37Disallow: /sponsor-content
38Disallow: /sponsored-content
39Disallow: /magazine/sponsor-content
40Disallow: /newsletters/politico-toplines
41
42User-agent: AddSearchBot
43Disallow: /
44
45User-agent: AI2Bot
46Disallow: /
47
48User-agent: Ai2Bot-Dolma
49Disallow: /
50
51User-agent: aiHitBot
52Disallow: /
53
54User-agent: Amazonbot
55Disallow: /
56
57User-agent: Andibot
58Disallow: /
59
60User-agent: anthropic-ai
61Disallow: /
62
63User-agent: Applebot
64Disallow: /
65
66User-agent: Applebot-Extended
67Disallow: /
68
69User-agent: Awario
70Disallow: /
71
72User-agent: bedrockbot
73Disallow: /
74
75User-agent: bigsur.ai
76Disallow: /
77
78User-agent: Brightbot 1.0
79Disallow: /
80
81User-agent: Bytespider
82Disallow: /
83
84User-agent: CCBot
85Disallow: /
86
87User-agent: ChatGPT Agent
88Disallow: /
89
90User-agent: ChatGPT-User
91Disallow: /
92
93User-agent: Claude-SearchBot
94Disallow: /
95
96User-agent: Claude-User
97Disallow: /
98
99User-agent: Claude-Web
100Disallow: /
101
102User-agent: ClaudeBot
103Disallow: /
104
105User-agent: CloudVertexBot
106Disallow: /
107
108User-agent: cohere-ai
109Disallow: /
110
111User-agent: cohere-training-data-crawler
112Disallow: /
113
114User-agent: Cotoyogi
115Disallow: /
116
117User-agent: Crawlspace
118Disallow: /
119
120User-agent: Datenbank Crawler
121Disallow: /
122
123User-agent: Devin
124Disallow: /
125
126User-agent: Diffbot
127Disallow: /
128
129User-agent: DuckAssistBot
130Disallow: /
131
132User-agent: Echobot Bot
133Disallow: /
134
135User-agent: EchoboxBot
136Disallow: /
137
138User-agent: FacebookBot
139Disallow: /
140
141User-agent: facebookexternalhit
142Disallow: /
143
144User-agent: Factset_spyderbot
145Disallow: /
146
147User-agent: FirecrawlAgent
148Disallow: /
149
150User-agent: FriendlyCrawler
151Disallow: /
152
153User-agent: Gemini-Deep-Research
154Disallow: /
155
156User-agent: Google-CloudVertexBot
157Disallow: /
158
159User-agent: Google-Extended
160Disallow: /
161
162User-agent: GoogleAgent-Mariner
163Disallow: /
164
165User-agent: GoogleOther
166Disallow: /
167
168User-agent: GoogleOther-Image
169Disallow: /
170
171User-agent: GoogleOther-Video
172Disallow: /
173
174User-agent: GPTBot
175Disallow: /
176
177User-agent: iaskspider/2.0
178Disallow: /
179
180User-agent: ICC-Crawler
181Disallow: /
182
183User-agent: ImagesiftBot
184Disallow: /
185
186User-agent: img2dataset
187Disallow: /
188
189User-agent: ISSCyberRiskCrawler
190Disallow: /
191
192User-agent: Kangaroo Bot
193Disallow: /
194
195User-agent: LinerBot
196Disallow: /
197
198User-agent: meta-externalagent
199Disallow: /
200
201User-agent: Meta-ExternalAgent
202Disallow: /
203
204User-agent: meta-externalfetcher
205Disallow: /
206
207User-agent: Meta-ExternalFetcher
208Disallow: /
209
210User-agent: MistralAI-User
211Disallow: /
212
213User-agent: MistralAI-User/1.0
214Disallow: /
215
216User-agent: MyCentralAIScraperBot
217Disallow: /
218
219User-agent: netEstate Imprint Crawler
220Disallow: /
221
222User-agent: NovaAct
223Disallow: /
224
225User-agent: OAI-SearchBot
226Disallow: /
227
228User-agent: omgili
229Disallow: /
230
231User-agent: omgilibot
232Disallow: /
233
234User-agent: Operator
235Disallow: /
236
237User-agent: PanguBot
238Disallow: /
239
240User-agent: Panscient
241Disallow: /
242
243User-agent: panscient.com
244Disallow: /
245
246User-agent: Perplexity-User
247Disallow: /
248
249User-agent: PerplexityBot
250Disallow: /
251
252User-agent: PetalBot
253Disallow: /
254
255User-agent: PhindBot
256Disallow: /
257
258User-agent: Poseidon Research Crawler
259Disallow: /
260
261User-agent: QualifiedBot
262Disallow: /
263
264User-agent: QuillBot
265Disallow: /
266
267User-agent: quillbot.com
268Disallow: /
269
270User-agent: SBIntuitionsBot
271Disallow: /
272
273User-agent: Scrapy
274Disallow: /
275
276User-agent: SemrushBot-OCOB
277Disallow: /
278
279User-agent: SemrushBot-SWA
280Disallow: /
281
282User-agent: Sidetrade indexer bot
283Disallow: /
284
285User-agent: Thinkbot
286Disallow: /
287
288User-agent: TikTokSpider
289Disallow: /
290
291User-agent: Timpibot
292Disallow: /
293
294User-agent: VelenPublicWebCrawler
295Disallow: /
296
297User-agent: WARDBot
298Disallow: /
299
300User-agent: Webzio-Extended
301Disallow: /
302
303User-agent: wpbot
304Disallow: /
305
306User-agent: YaK
307Disallow: /
308
309User-agent: YandexAdditional
310Disallow: /
311
312User-agent: YandexAdditionalBot
313Disallow: /
314
315User-agent: YouBot
316Disallow: /