Axios / robots.txt
← back to axios.com · change detected 28d ago (2026-07-07T20:35:12Z) · modified +8 −0
What changed
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
Diff old (2026-06-20T01:10:30Z) → new (2026-07-07T20:35:12Z)
| @@ -4,6 +4,14 @@ | ||
| 4 | 4 | User-agent: Twitterbot/* |
| 5 | 5 | Disallow: |
| 6 | 6 | Crawl-delay: 1 |
| 7 | + | |
| 8 | + User-agent: Google-Extended | |
| 9 | + Allow: / | |
| 10 | + Disallow: /core/* | |
| 11 | + Disallow: /admin/ | |
| 12 | + Disallow: /login/ | |
| 13 | + Disallow: /sso/ | |
| 14 | + Disallow: /api/ | |
| 7 | 15 | |
| 8 | 16 | User-agent: * |
| 9 | 17 | Disallow: /core/* |