NeuralCrawl

Allstate / robots.txt

← back to allstate.com · change detected 1mo ago (2026-10-01T01:22:45Z) · modified +17 −4

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-10-01T01:22:45Z)

@@ -1,12 +1,25 @@
11  User-Agent: *
2- Sitemap: https://www.allstate.com/sitemap.xml
32  Disallow: /_test*
3+ Disallow: /allstate-benefits/*
44  Disallow: /credit
5+ Disallow: /ddt-test-pages*
56  Disallow: /landingpages*
7+ Disallow: /personal
68  Disallow: /qat*
7- Disallow: /resources/allstate/attachments/pdf/etr0509.pdf
9+ Disallow: /resources/allstate/attachments/*
810  Disallow: /resources/resourcetest
911  Disallow: /staff-counsel*
1012  Disallow: /test*
11- Disallow: /tests*
12- Disallow: /api/
13+ Disallow: /es/_test*
14+ Disallow: /es/allstate-benefits/*
15+ Disallow: /es/auto-insurance/cars-trucks
16+ Disallow: /es/credit
17+ Disallow: /es/ddt-test-pages*
18+ Disallow: /es/landingpages*
19+ Disallow: /es/personal
20+ Disallow: /es/qat*
21+ Disallow: /es/resources/allstate/attachments/*
22+ Disallow: /es/resources/resourcetest
23+ Disallow: /es/staff-counsel*
24+ Disallow: /es/test*
25+ Sitemap: https://www.allstate.com/sitemap.xml