The Hill / robots.txt
← back to thehill.com · change detected 16d ago (2026-07-21T19:32:39Z) · modified +3 −0
What changed
- Added rules for user-agent ‘AmazonAdBot’
Diff old (2026-06-20T01:10:30Z) → new (2026-07-21T19:32:39Z)
| @@ -30,6 +30,9 @@ | ||
| 30 | 30 | User-Agent: PerplexityBot |
| 31 | 31 | Disallow: / |
| 32 | 32 | Allow: /advertise/ |
| 33 | + | |
| 34 | + User-agent: AmazonAdBot | |
| 35 | + Allow: / | |
| 33 | 36 | |
| 34 | 37 | User-agent: ia_archiver |
| 35 | 38 | Disallow: / |