Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 466.321721ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:40:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.352671ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 423.302115ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:39:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.359712ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 500.211792ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:38:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 500.237099ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 454.657453ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:37:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 454.684198ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 463.282493ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:36:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 463.305421ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 444.390386ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:35:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.42412ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 455.271602ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:34:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.300239ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 529.267621ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:34:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.297325ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 726.337697ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:33:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 726.424069ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 489.431264ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:33:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.456047ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 702.773464ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:33:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 702.801573ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 565.488693ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:32:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 565.513135ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 489.893299ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:32:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.915893ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 518.496383ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:32:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.526291ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 434.091193ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.112815ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 513.668117ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:31:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 513.695485ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 541.594355ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:31:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 541.628329ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 504.853569ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:30:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 505.198173ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 476.30787ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:30:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.329629ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 466.059346ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:30:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.100499ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 484.250174ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:29:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.283604ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 571.464763ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.508982ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 482.134477ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:28:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.161851ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 482.968862ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:28:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.992012ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 410.405358ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:27:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 410.452803ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 614.560583ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:27:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 614.667003ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 486.124364ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.141651ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 505.740207ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:27:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 505.76843ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 585.249905ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:26:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 585.275848ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 456.546477ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:26:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.590871ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 491.268554ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:25:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.29672ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 483.547731ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:25:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.571187ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 702.195804ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:25:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 702.228126ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 470.801466ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:24:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.834368ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 454.388627ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:24:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 454.416554ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 456.27894ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:23:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.308855ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 449.369751ms trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:23:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 449.391432ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000633619s trace: https://jaeger.dev.storj.io/trace/597cf334d09f493d FAIL (2025-12-18 18:23:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000655841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 469.704159ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:22:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.754297ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 457.982702ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:22:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.000245ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 504.338061ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:21:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 504.370039ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 482.037906ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:20:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.118723ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 427.124785ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:19:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.148805ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 446.351051ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:19:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.379228ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 468.242104ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:18:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 468.26928ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 458.889987ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.920176ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 484.214345ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:17:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 484.246573ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 446.063575ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:17:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.096365ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 524.487997ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:16:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 524.536693ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 507.298279ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:15:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 507.392033ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)