Axios / robots.txt snapshot

← back to axios.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 521 bytes · sha256 904dc30a14226f38 · raw

final URL: https://www.axios.com/robots.txt

1
2	# Disallow Rules
3
4	User-agent: Twitterbot/*
5	Disallow:
6	Crawl-delay: 1
7
8	User-agent: *
9	Disallow: /core/*
10	Disallow: /admin/
11	Disallow: /login/
12	Disallow: /sso/
13	Disallow: /api/
14
15	# Disallow Rules (bot-specific)
16
17	User-agent: CCBot
18	Disallow: /
19
20	User-agent: Bytespider
21	Disallow: /
22
23	User-agent: Diffbot
24	Disallow: /
25
26	User-agent: FacebookBot
27	Disallow: /
28
29	User-agent: ImagesiftBot
30	Disallow: /
31
32	User-agent: Scrapy
33	Disallow: /
34
35	User-agent: Amazonbot
36	Disallow: /
37
38	# Sitemaps
39
40	Sitemap: https://www.axios.com/sitemap.xml
41
42	# 8fkux4mqab196bbs