Nestle / robots.txt
← back to nestle.com · change detected 9d ago (2026-07-28T03:59:04Z) · modified +2 −0
What changed
- Formatting/comment-only change (no semantic difference)
Diff old (2026-06-20T01:10:30Z) → new (2026-07-28T03:59:04Z)
| @@ -14,6 +14,8 @@ | ||
| 14 | 14 | # http://www.robotstxt.org/robotstxt.html |
| 15 | 15 | |
| 16 | 16 | User-agent: * |
| 17 | + # Content-Signal: declares how crawlers may use this content - ai-train=no (no AI model training), search=yes (allow search indexing), ai-input=yes (allow use as input for AI features). | |
| 18 | + Content-Signal: ai-train=no, search=yes, ai-input=yes | |
| 17 | 19 | # CSS, JS, Images |
| 18 | 20 | Allow: /core/*.css$ |
| 19 | 21 | Allow: /core/*.css? |