The Hill / robots.txt
← back to thehill.com · change detected 2mo ago (2026-08-11T21:11:35Z) · modified +3 −0
What changed
- Added rules for user-agent ‘carbon-umbrella-bot’
Diff old (2026-07-21T19:32:39Z) → new (2026-08-11T21:11:35Z)
| @@ -32,6 +32,9 @@ | ||
| 32 | 32 | Allow: /advertise/ |
| 33 | 33 | |
| 34 | 34 | User-agent: AmazonAdBot |
| 35 | + Allow: / | |
| 36 | + | |
| 37 | + User-agent: carbon-umbrella-bot | |
| 35 | 38 | Allow: / |
| 36 | 39 | |
| 37 | 40 | User-agent: ia_archiver |