Colgate-Palmolive / robots.txt
← back to colgatepalmolive.com · change detected 18d ago (2026-07-19T06:36:01Z) · modified +3 −10
What changed
- Added Allow: / for *
- Removed Disallow: /*?brand= for *
- Removed Disallow: /*?condition= for *
- Removed Disallow: /*?lifestage= for *
- Removed Disallow: /*?lightboxfired= for *
- Removed Disallow: /*?productform= for *
- Removed Disallow: /*?searchTerm= for *
- Crawl-delay for *: 10 → none
- Added sitemap /sitemap.xml
- Removed sitemap https://www.colgatepalmolive.com/en-us/sitemap.xml
Diff old (2026-06-20T01:10:31Z) → new (2026-07-19T06:36:01Z)
| @@ -1,11 +1,4 @@ | ||
| 1 | - Crawl-delay: 10 | |
| 2 | - | |
| 3 | 1 | User-agent: * |
| 4 | - Disallow: /*?searchTerm= | |
| 5 | - Disallow: /*?condition= | |
| 6 | - Disallow: /*?lifestage= | |
| 7 | - Disallow: /*?productform= | |
| 8 | - Disallow: /*?brand= | |
| 9 | - Disallow: /*?lightboxfired= | |
| 10 | - | |
| 11 | - Sitemap: https://www.colgatepalmolive.com/en-us/sitemap.xml | |
| 2 | + Allow: / | |
| 3 | + LLM-Policy: /llms.txt | |
| 4 | + Sitemap: /sitemap.xml |