Colgate-Palmolive / robots.txt
← back to colgatepalmolive.com · change detected 1mo ago (2026-09-25T11:16:20Z) · modified +11 −2
What changed
- Added rules for user-agent ‘ColPal-Elastic-Crawler’
- Added rules for user-agent ‘Elastic-Crawler’
- Added rules for user-agent ‘Swiftbot’
- Added sitemap https://www.colgatepalmolive.com/en-us/sitemap.xml
- Added sitemap https://www.colgatepalmolive.com/et-ee/sitemap.xml
- Removed sitemap /sitemap.xml
Diff old (2026-07-19T06:36:01Z) → new (2026-09-25T11:16:20Z)
| @@ -1,4 +1,13 @@ | ||
| 1 | + User-agent: Swiftbot | |
| 2 | + User-agent: Elastic-Crawler | |
| 3 | + User-agent: ColPal-Elastic-Crawler | |
| 4 | + Allow: / | |
| 5 | + Crawl-delay: 5 | |
| 6 | + | |
| 7 | + | |
| 1 | 8 | User-agent: * |
| 2 | 9 | Allow: / |
| 3 | - LLM-Policy: /llms.txt | |
| 4 | - Sitemap: /sitemap.xml | |
| 10 | + | |
| 11 | + | |
| 12 | + Sitemap: https://www.colgatepalmolive.com/en-us/sitemap.xml | |
| 13 | + Sitemap: https://www.colgatepalmolive.com/et-ee/sitemap.xml |