System status
Health of the 24/7 archiving pipeline.
1273companies
1198archived
2530snapshots
1357changes
Recent crawl runs
| Started | Finished | Total | OK | Failed | Changed |
|---|---|---|---|---|---|
| 2026-08-04T18:41:50Z | 2026-08-04T18:49:16Z | 1273 | 1154 | 119 | 12 |
| 2026-08-04T12:35:21Z | 2026-08-04T12:42:22Z | 1273 | 1153 | 120 | 14 |
| 2026-08-04T06:27:31Z | 2026-08-04T06:35:30Z | 1273 | 1156 | 117 | 11 |
| 2026-08-04T00:20:43Z | 2026-08-04T00:27:51Z | 1273 | 1151 | 122 | 9 |
| 2026-08-03T18:20:01Z | 2026-08-03T18:27:05Z | 1273 | 1154 | 119 | 7 |
| 2026-08-03T12:14:42Z | 2026-08-03T12:21:46Z | 1273 | 1155 | 118 | 7 |
| 2026-08-03T06:05:34Z | 2026-08-03T06:12:38Z | 1273 | 1155 | 118 | 5 |
| 2026-08-03T00:01:01Z | 2026-08-03T00:07:57Z | 1273 | 1142 | 131 | 4 |
| 2026-08-02T17:54:01Z | 2026-08-02T18:01:38Z | 1273 | 1154 | 119 | 6 |
| 2026-08-02T11:50:56Z | 2026-08-02T11:58:11Z | 1273 | 1156 | 117 | 7 |
| 2026-08-02T05:48:35Z | 2026-08-02T05:55:30Z | 1273 | 1153 | 120 | 3 |
| 2026-08-01T23:44:31Z | 2026-08-01T23:51:41Z | 1273 | 1153 | 120 | 6 |
| 2026-08-01T17:37:01Z | 2026-08-01T17:43:59Z | 1273 | 1150 | 123 | 6 |
| 2026-08-01T11:32:01Z | 2026-08-01T11:39:00Z | 1273 | 1154 | 119 | 8 |
| 2026-08-01T05:23:42Z | 2026-08-01T05:30:37Z | 1273 | 1154 | 119 | 7 |
| 2026-07-31T23:15:13Z | 2026-07-31T23:22:14Z | 1273 | 1151 | 122 | 9 |
| 2026-07-31T17:09:57Z | 2026-07-31T17:16:59Z | 1273 | 1153 | 120 | 13 |
| 2026-07-31T11:01:49Z | 2026-07-31T11:09:44Z | 1273 | 1153 | 120 | 6 |
| 2026-07-31T04:57:50Z | 2026-07-31T05:05:07Z | 1273 | 1153 | 120 | 6 |
| 2026-07-30T22:54:00Z | 2026-07-30T23:00:57Z | 1273 | 1152 | 121 | 10 |
Companies with repeated fetch failures
No archived robots.txt and 2+ consecutive failed fetches.
| Company | Domain | Failures | HTTP response | Last error | Last checked |
|---|---|---|---|---|---|
| BJ's Wholesale Club | bjs.com | 187 | 403 | HTTP 403 | 4h ago |
| Bunge | bunge.com | 187 | 403 | HTTP 403 | 4h ago |
| Carvana | carvana.com | 187 | 403 | HTTP 403 | 4h ago |
| Casey's General Stores | caseys.com | 187 | 403 | HTTP 403 | 4h ago |
| China Construction Bank | ccb.com | 187 | — | ConnectError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disa | 4h ago |
| CDW | cdw.com | 187 | — | RemoteProtocolError: <StreamReset stream_id:1, error_code:2, remote_reset:True> | 4h ago |
| CSX | csx.com | 187 | 403 | HTTP 403 | 4h ago |
| D.R. Horton | drhorton.com | 187 | 403 | HTTP 403 | 4h ago |
| Energy Transfer | energytransfer.com | 187 | 403 | HTTP 403 | 4h ago |
| EOG Resources | eogresources.com | 187 | 200 | Automated fetch: HTML response, not robots.txt | 4h ago |
| Group 1 Automotive | group1auto.com | 187 | 403 | HTTP 403 | 4h ago |
| Hanesbrands | hbi.com | 187 | 200 | Automated fetch: Redirect (2 redirects): https://www.hbi.com/robots.txt → https://hubbardb | 4h ago |
| HDFC Bank | hdfcbank.com | 187 | 403 | HTTP 403 | 4h ago |
| Hearst | hearst.com | 187 | 403 | HTTP 403 | 4h ago |
| Inditex | inditex.com | 187 | 200 | Automated fetch: Redirect (3 redirects): https://www.inditex.com/robots.txt → https://www. | 4h ago |
| JBS | jbs.com.br | 187 | 403 | HTTP 403 | 4h ago |
| Macmillan | macmillan.com | 187 | — | ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has | 4h ago |
| Maybank | maybank.com | 187 | — | ReadTimeout: | 4h ago |
| Paychex | paychex.com | 187 | 403 | HTTP 403 | 4h ago |
| Reliance Industries | ril.com | 187 | 200 | Automated fetch: HTML response, not robots.txt | 4h ago |
| T-Mobile US | t-mobile.com | 187 | 403 | HTTP 403 | 4h ago |
| Tidal | tidal.com | 187 | 403 | HTTP 403 | 4h ago |
| Ulta Beauty | ulta.com | 187 | 403 | HTTP 403 | 4h ago |
| Wakefern Food | wakefern.com | 187 | — | ConnectError: All connection attempts failed | 4h ago |
| Zalando | zalando.com | 187 | 200 | Automated fetch: HTML response, not robots.txt | 4h ago |
| Bank of China | boc.cn | 182 | 200 | Automated fetch: Redirect (1 redirect): https://www.boc.cn/robots.txt → https://www.boc.cn | 4h ago |
| CATL | catl.com | 182 | 403 | HTTP 403 | 4h ago |
| Mandiant | mandiant.com | 182 | 200 | Automated fetch: Redirect (2 redirects): https://www.mandiant.com/robots.txt → https://clo | 4h ago |
| Live | live.com | 157 | 200 | Automated fetch: Redirect (1 redirect): https://www.live.com/robots.txt → https://outlook. | 4h ago |
| Office | office.com | 157 | 200 | Automated fetch: Redirect (1 redirect): https://www.office.com/robots.txt → https://login. | 4h ago |
| Shopee | shopee.com | 157 | 200 | Automated fetch: Redirect (1 redirect): https://www.shopee.com/robots.txt → https://shopee | 4h ago |
| HAL Open Archive | hal.science | 156 | 200 | Automated fetch: Redirect (2 redirects): https://www.hal.science/robots.txt → https://hal. | 4h ago |
| National University of Singapore | nus.edu.sg | 156 | 200 | Automated fetch: HTML response, not robots.txt | 4h ago |