System status
Health of the 24/7 archiving pipeline.
1273companies
1204archived
3937snapshots
2782changes
Recent crawl runs
| Started | Finished | Total | OK | Failed | Changed |
|---|---|---|---|---|---|
| 2026-09-19T09:26:31Z | 2026-09-19T09:33:52Z | 1273 | 1149 | 124 | 6 |
| 2026-09-19T03:26:01Z | 2026-09-19T03:33:07Z | 1273 | 1138 | 135 | 3 |
| 2026-09-18T21:24:06Z | 2026-09-18T21:31:03Z | 1273 | 1149 | 124 | 8 |
| 2026-09-18T15:20:16Z | 2026-09-18T15:27:37Z | 1273 | 1150 | 123 | 7 |
| 2026-09-18T09:12:01Z | 2026-09-18T09:19:18Z | 1273 | 1151 | 122 | 8 |
| 2026-09-18T03:07:21Z | 2026-09-18T03:14:41Z | 1273 | 1151 | 122 | 5 |
| 2026-09-17T21:06:01Z | 2026-09-17T21:12:52Z | 1273 | 1148 | 125 | 12 |
| 2026-09-17T15:04:01Z | 2026-09-17T15:11:14Z | 1273 | 1148 | 125 | 10 |
| 2026-09-17T09:03:23Z | 2026-09-17T09:10:48Z | 1273 | 1149 | 124 | 5 |
| 2026-09-17T02:54:51Z | 2026-09-17T03:02:10Z | 1273 | 1143 | 130 | 7 |
| 2026-09-16T20:51:56Z | 2026-09-16T20:58:51Z | 1273 | 1137 | 136 | 9 |
| 2026-09-16T14:44:20Z | 2026-09-16T14:51:21Z | 1273 | 1149 | 124 | 6 |
| 2026-09-16T08:38:14Z | 2026-09-16T08:45:16Z | 1273 | 1146 | 127 | 6 |
| 2026-09-16T02:35:53Z | 2026-09-16T02:43:17Z | 1273 | 1147 | 126 | 16 |
| 2026-09-15T20:34:01Z | 2026-09-15T20:40:55Z | 1273 | 1153 | 120 | 9 |
| 2026-09-15T14:27:01Z | 2026-09-15T14:34:35Z | 1273 | 1155 | 118 | 12 |
| 2026-09-15T08:19:01Z | 2026-09-15T08:26:44Z | 1273 | 1150 | 123 | 7 |
| 2026-09-15T02:16:00Z | 2026-09-15T02:23:28Z | 1273 | 1151 | 122 | 7 |
| 2026-09-14T20:07:01Z | 2026-09-14T20:14:08Z | 1273 | 1151 | 122 | 9 |
| 2026-09-14T14:05:01Z | 2026-09-14T14:12:31Z | 1273 | 1151 | 122 | 10 |
Companies with repeated fetch failures
No archived robots.txt and 2+ consecutive failed fetches.
| Company | Domain | Failures | HTTP response | Last error | Last checked |
|---|---|---|---|---|---|
| BJ's Wholesale Club | bjs.com | 367 | 403 | HTTP 403 | 1h ago |
| Bunge | bunge.com | 367 | 403 | HTTP 403 | 1h ago |
| Carvana | carvana.com | 367 | 403 | HTTP 403 | 1h ago |
| China Construction Bank | ccb.com | 367 | — | ConnectError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disa | 1h ago |
| CDW | cdw.com | 367 | — | RemoteProtocolError: <StreamReset stream_id:1, error_code:2, remote_reset:True> | 1h ago |
| CSX | csx.com | 367 | 403 | HTTP 403 | 1h ago |
| D.R. Horton | drhorton.com | 367 | 403 | HTTP 403 | 1h ago |
| Energy Transfer | energytransfer.com | 367 | 403 | HTTP 403 | 1h ago |
| EOG Resources | eogresources.com | 367 | 200 | Automated fetch: HTML response, not robots.txt | 1h ago |
| Group 1 Automotive | group1auto.com | 367 | 403 | HTTP 403 | 1h ago |
| Hanesbrands | hbi.com | 367 | 200 | Automated fetch: Redirect (2 redirects): https://www.hbi.com/robots.txt → https://hubbardb | 1h ago |
| HDFC Bank | hdfcbank.com | 367 | 403 | HTTP 403 | 1h ago |
| Hearst | hearst.com | 367 | 403 | HTTP 403 | 1h ago |
| Inditex | inditex.com | 367 | 200 | Automated fetch: Redirect (3 redirects): https://www.inditex.com/robots.txt → https://www. | 1h ago |
| JBS | jbs.com.br | 367 | 403 | HTTP 403 | 1h ago |
| Macmillan | macmillan.com | 367 | — | ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has | 1h ago |
| Maybank | maybank.com | 367 | 503 | ReadTimeout: | 1h ago |
| Paychex | paychex.com | 367 | 403 | HTTP 403 | 1h ago |
| Reliance Industries | ril.com | 367 | 200 | Automated fetch: HTML response, not robots.txt | 1h ago |
| T-Mobile US | t-mobile.com | 367 | 403 | HTTP 403 | 1h ago |
| Ulta Beauty | ulta.com | 367 | 403 | HTTP 403 | 1h ago |
| Wakefern Food | wakefern.com | 367 | — | ConnectError: All connection attempts failed | 1h ago |
| Zalando | zalando.com | 367 | 200 | Automated fetch: HTML response, not robots.txt | 1h ago |
| Bank of China | boc.cn | 362 | 200 | Automated fetch: Redirect (1 redirect): https://www.boc.cn/robots.txt → https://www.boc.cn | 1h ago |
| CATL | catl.com | 362 | 200 | Automated fetch: Redirect (2 redirects): https://www.catl.com/robots.txt → https://www.cat | 1h ago |
| Mandiant | mandiant.com | 362 | 200 | Automated fetch: Redirect (2 redirects): https://www.mandiant.com/robots.txt → https://clo | 1h ago |
| Live | live.com | 337 | 200 | Automated fetch: Redirect (1 redirect): https://www.live.com/robots.txt → https://outlook. | 1h ago |
| Office | office.com | 337 | 200 | Automated fetch: Redirect (1 redirect): https://www.office.com/robots.txt → https://login. | 1h ago |
| Shopee | shopee.com | 337 | 200 | Automated fetch: Redirect (1 redirect): https://www.shopee.com/robots.txt → https://shopee | 1h ago |
| HAL Open Archive | hal.science | 336 | 200 | Automated fetch: Redirect (2 redirects): https://www.hal.science/robots.txt → https://hal. | 1h ago |
| National University of Singapore | nus.edu.sg | 336 | 200 | Automated fetch: HTML response, not robots.txt | 1h ago |
| Condé Nast | condenast.com | 75 | 403 | HTTP 403 | 1h ago |