System status
Health of the 24/7 archiving pipeline.
1273companies
1206archived
5416snapshots
4280changes
Recent crawl runs
| Started | Finished | Total | OK | Failed | Changed |
|---|---|---|---|---|---|
| 2026-11-07T08:12:01Z | 2026-11-07T08:19:09Z | 1273 | 1143 | 130 | 7 |
| 2026-11-07T02:10:01Z | 2026-11-07T02:17:01Z | 1273 | 1136 | 137 | 6 |
| 2026-11-06T20:07:01Z | 2026-11-06T20:13:58Z | 1273 | 1136 | 137 | 14 |
| 2026-11-06T13:58:38Z | 2026-11-06T14:05:37Z | 1273 | 1144 | 129 | 9 |
| 2026-11-06T07:53:23Z | 2026-11-06T08:00:37Z | 1273 | 1143 | 130 | 8 |
| 2026-11-06T01:51:48Z | 2026-11-06T01:59:06Z | 1273 | 1143 | 130 | 8 |
| 2026-11-05T19:46:51Z | 2026-11-05T19:53:48Z | 1273 | 1139 | 134 | 16 |
| 2026-11-05T13:40:39Z | 2026-11-05T13:47:51Z | 1273 | 1143 | 130 | 10 |
| 2026-11-05T07:38:01Z | 2026-11-05T07:45:14Z | 1273 | 1144 | 129 | 10 |
| 2026-11-05T01:31:01Z | 2026-11-05T01:38:00Z | 1273 | 1141 | 132 | 7 |
| 2026-11-04T19:27:01Z | 2026-11-04T19:33:55Z | 1273 | 1143 | 130 | 12 |
| 2026-11-04T13:25:01Z | 2026-11-04T13:32:12Z | 1273 | 1143 | 130 | 9 |
| 2026-11-04T07:17:30Z | 2026-11-04T07:25:22Z | 1273 | 1142 | 131 | 6 |
| 2026-11-04T01:12:44Z | 2026-11-04T01:20:07Z | 1273 | 1131 | 142 | 10 |
| 2026-11-03T19:03:01Z | 2026-11-03T19:10:24Z | 1273 | 1140 | 133 | 16 |
| 2026-11-03T12:53:01Z | 2026-11-03T13:00:12Z | 1273 | 1142 | 131 | 8 |
| 2026-11-03T06:52:26Z | 2026-11-03T06:59:34Z | 1273 | 1143 | 130 | 6 |
| 2026-11-03T00:50:51Z | 2026-11-03T00:58:15Z | 1273 | 1139 | 134 | 6 |
| 2026-11-02T18:48:01Z | 2026-11-02T18:55:02Z | 1273 | 1140 | 133 | 11 |
| 2026-11-02T12:46:41Z | 2026-11-02T12:53:58Z | 1273 | 1141 | 132 | 8 |
Companies with repeated fetch failures
No archived robots.txt and 2+ consecutive failed fetches.
| Company | Domain | Failures | HTTP response | Last error | Last checked |
|---|---|---|---|---|---|
| BJ's Wholesale Club | bjs.com | 560 | 403 | HTTP 403 | 5h ago |
| Bunge | bunge.com | 560 | 403 | HTTP 403 | 5h ago |
| Carvana | carvana.com | 560 | 403 | HTTP 403 | 5h ago |
| China Construction Bank | ccb.com | 560 | — | ConnectError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disa | 5h ago |
| CDW | cdw.com | 560 | 403 | HTTP 403 | 5h ago |
| CSX | csx.com | 560 | 403 | HTTP 403 | 5h ago |
| D.R. Horton | drhorton.com | 560 | 403 | HTTP 403 | 5h ago |
| Energy Transfer | energytransfer.com | 560 | 403 | HTTP 403 | 5h ago |
| Group 1 Automotive | group1auto.com | 560 | 403 | HTTP 403 | 5h ago |
| Hanesbrands | hbi.com | 560 | 200 | Automated fetch: Redirect (2 redirects): https://www.hbi.com/robots.txt → https://hubbardb | 5h ago |
| HDFC Bank | hdfcbank.com | 560 | 403 | HTTP 403 | 5h ago |
| Hearst | hearst.com | 560 | 403 | HTTP 403 | 5h ago |
| Inditex | inditex.com | 560 | 200 | Automated fetch: Redirect (3 redirects): https://www.inditex.com/robots.txt → https://www. | 5h ago |
| Macmillan | macmillan.com | 560 | — | ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has | 5h ago |
| Maybank | maybank.com | 560 | 503 | ReadTimeout: | 5h ago |
| Paychex | paychex.com | 560 | 403 | HTTP 403 | 5h ago |
| Reliance Industries | ril.com | 560 | 200 | Automated fetch: HTML response, not robots.txt | 5h ago |
| T-Mobile US | t-mobile.com | 560 | 403 | HTTP 403 | 5h ago |
| Ulta Beauty | ulta.com | 560 | 403 | HTTP 403 | 5h ago |
| Wakefern Food | wakefern.com | 560 | — | ConnectError: All connection attempts failed | 5h ago |
| Zalando | zalando.com | 560 | 200 | Automated fetch: HTML response, not robots.txt | 5h ago |
| Bank of China | boc.cn | 555 | 200 | Automated fetch: Redirect (1 redirect): https://www.boc.cn/robots.txt → https://www.boc.cn | 5h ago |
| CATL | catl.com | 555 | 200 | Automated fetch: Redirect (2 redirects): https://www.catl.com/robots.txt → https://www.cat | 5h ago |
| Mandiant | mandiant.com | 555 | 200 | Automated fetch: Redirect (2 redirects): https://www.mandiant.com/robots.txt → https://clo | 5h ago |
| Live | live.com | 530 | 200 | Automated fetch: Redirect (1 redirect): https://www.live.com/robots.txt → https://outlook. | 5h ago |
| Office | office.com | 530 | 200 | Automated fetch: Redirect (1 redirect): https://www.office.com/robots.txt → https://login. | 5h ago |
| Shopee | shopee.com | 530 | 500 | HTTP 500 | 5h ago |
| HAL Open Archive | hal.science | 529 | 200 | Automated fetch: Redirect (2 redirects): https://www.hal.science/robots.txt → https://hal. | 5h ago |
| National University of Singapore | nus.edu.sg | 529 | 200 | Automated fetch: HTML response, not robots.txt | 5h ago |
| Indonesia (Indonesia.go.id) | indonesia.go.id | 150 | 404 | ConnectError: [Errno -2] Name or service not known | 5h ago |
| Condé Nast | condenast.com | 84 | 403 | HTTP 403 | 5h ago |