C.H. Robinson / robots.txt
← back to chrobinson.com · change detected 1mo ago (2026-09-25T23:27:23Z) · modified +6 −0
What changed
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- Added Disallow: Amazonbot for *
- Added Disallow: Amzn-SearchBot for *
- Added Disallow: Bytespider for *
Diff old (2026-06-20T01:10:30Z) → new (2026-09-25T23:27:23Z)
| @@ -35,6 +35,8 @@ | ||
| 35 | 35 | User-agent: omgili |
| 36 | 36 | User-agent: TimpiBot |
| 37 | 37 | User-agent: MistralAI-User |
| 38 | + User-agent: Claude-User | |
| 39 | + User-agent: Claude-SearchBot | |
| 38 | 40 | Allow: / |
| 39 | 41 | |
| 40 | 42 | |
| @@ -78,6 +80,10 @@ | ||
| 78 | 80 | Disallow: *g-xsm* |
| 79 | 81 | Disallow: *wxc-gs1* |
| 80 | 82 | Disallow: *wxc-gs2* |
| 83 | + Disallow: Bytespider | |
| 84 | + Disallow: Amazonbot | |
| 85 | + Disallow: Amzn-SearchBot | |
| 86 | + | |
| 81 | 87 | |
| 82 | 88 | |
| 83 | 89 | Sitemap: https://www.chrobinson.com/chrobinson-de-de-sitemap.xml |