Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 466.5251ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:15:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.54998ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 483.634637ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:14:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.666322ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 479.809ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:13:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.836834ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 486.213487ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:12:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 486.240065ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 561.263331ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:11:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 561.422712ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 565.828806ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 565.880164ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 519.993332ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 520.020879ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 489.339105ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:09:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.369338ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 491.503794ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:09:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.540568ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 486.804833ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 486.833582ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 482.294092ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.413582ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 463.331612ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:07:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.358119ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 470.655476ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:06:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.685943ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 501.835687ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:05:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.863945ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 491.169165ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.195274ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 545.02826ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 545.054089ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 487.923584ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.960627ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 493.068375ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.111196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 560.451517ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:03:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 560.497221ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 507.513364ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 507.54572ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 476.228567ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:01:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.254069ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 473.848811ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:01:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.892233ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 498.944798ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 18:00:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 498.978398ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 452.535524ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:59:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.561882ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 508.88048ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:59:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 508.908698ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 537.120729ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 537.151678ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 512.583406ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:58:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 512.608657ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 462.252761ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.286702ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 506.187752ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.217438ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 553.634683ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:56:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.652148ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 476.349357ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:54:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.367773ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 584.187222ms trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 584.216385ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.931030711s trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:53:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.931057282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.031534929s trace: https://jaeger.dev.storj.io/trace/287bcf3f0b56b1a5 FAIL (2025-12-18 17:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 18 17:52:24.492435493 (314h24m25.320143887s) [upload=327057]: Starting upload Dec 18 17:52:24.492526044 (314h24m25.320234438s) [upload=327057]: Done starting upload Dec 18 17:52:24.492690492 (314h24m25.320398875s) [upload=327057]: Uploading segments... Dec 18 17:52:24.492997272 (314h24m25.320705664s) [upload=327057]: Got next segment. Inline: false Dec 18 17:52:24.493045113 (314h24m25.320753494s) [upload=327057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 17:52:24.493053379 (314h24m25.32076176s) [upload=327057]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 17:52:24.521958937 (314h24m25.349667328s) [seg_pos={0 0}, upload=327057]: Begin upload segment... Dec 18 17:52:24.522020538 (314h24m25.349728915s) [seg_pos={0 0}, upload=327057]: Done begin upload segment. Dec 18 17:52:24.522070965 (314h24m25.349779356s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522092237 (314h24m25.349800639s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522821125 (314h24m25.350529519s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.522952363 (314h24m25.350660765s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.523691581 (314h24m25.351399978s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524197796 (314h24m25.351906195s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.524621838 (314h24m25.352330236s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525142863 (314h24m25.352851255s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.525275874 (314h24m25.352984256s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526274960 (314h24m25.353983342s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.526713718 (314h24m25.354422109s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527171413 (314h24m25.354879804s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527533620 (314h24m25.355242014s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.527900269 (314h24m25.35560866s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528324561 (314h24m25.356032954s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.528724248 (314h24m25.356432651s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Uploading piece... Dec 18 17:52:24.593836701 (314h24m25.421545098s) [noise=true , address=145.79.227.41:4087 , piece_id=COD47SMEXMI42X6DRE3IMOCJ75PY7KN4TWJ3PIFMBALBFWRPBCOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: storage node overloaded, request limit: 2000 Dec 18 17:52:24.612773243 (314h24m25.440481627s) [upload=327057]: Next returned nil segment Dec 18 17:52:24.612802799 (314h24m25.440511191s) [upload=327057]: Waiting for error group managing segments... Dec 18 17:52:24.617899015 (314h24m25.445607415s) [noise=true , address=45.91.116.6:4973 , piece_id=YPA6EBELOF3L3LQTAJU33LYW4BBU4O4RXCBGSZBZM4PW26XX5TEA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.620519311 (314h24m25.448227702s) [noise=true , address=77.90.31.17:4773 , piece_id=XALU2HKPRA7FHFAT6UULGMDJSTFK7CCNIMK5U76FIBJ7OLKJIGIA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.621686616 (314h24m25.449395015s) [noise=true , address=45.91.116.12:4228 , piece_id=33LYSKXAV5TB6VP5OON3276WTMMPZLN35NJBMCUVAULCYYOEL73A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.622753052 (314h24m25.450461444s) [noise=true , address=145.79.227.49:4338 , piece_id=YWLBOLVBK5UDYUKGKEQXSIUI6EQBULJGE4B4TFKTIOTHXQ23IR2A, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623312295 (314h24m25.451020693s) [noise=true , address=145.79.227.50:4373 , piece_id=3EA6BLHE6AROZPXEMGBX7KZPMWOD444JFZXAKAR7Y54T56JLAZBA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.623402048 (314h24m25.451110454s) [noise=true , address=145.79.227.48:4303 , piece_id=CKCRIJQHAXCESFIW6MTXHF47PB2QUHSIQTDYQWWPHQWJA7CALQAQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.624554737 (314h24m25.452263139s) [noise=true , address=145.79.227.33:4869 , piece_id=G4563EJHI7X7L7ABVG7GDVKDHJME435MDSXX7VUOLKJJCNKJAYCQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.625310754 (314h24m25.453019148s) [noise=true , address=145.79.227.31:4797 , piece_id=ULKKI6QDOGGRFXMMQ2VJMA2CY5XYXDJI7VIZKXQUWZROGBIS3BNQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.629367553 (314h24m25.457075947s) [noise=true , address=77.90.31.13:4628 , piece_id=YNTDXYVRZWZHMVYTLPYK6QCWHDDIEHJKPENHHFCLR7WKOJBQT6TQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.630559175 (314h24m25.458267567s) [noise=true , address=45.91.116.8:4048 , piece_id=D3KM3VF53NDPHAP5B42HURRU3JPBNZNDXSAKRE3WYYZAHNM3NFOA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.635968007 (314h24m25.463676389s) [noise=true , address=145.79.227.37:4565 , piece_id=XPWZCIICRRCBOSUUOVNZIJYNPHRTNEE6D3POQLBAKC23FTE6U67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:24.638397380 (314h24m25.466105775s) [noise=true , address=145.79.227.52:4446 , piece_id=37TJHZY2RIAQKXTUPYCKQR37LMIG32PMLPAQRGNU7LUQT7YKOWVQ, seg_pos={0 0}, upload=327057]: Done uploading piece. err: <nil> Dec 18 17:52:54.462410380 (314h24m55.29011879s) [noise=true , address=145.79.227.28:4726 , piece_id=V3GFT6OG7QG7HCJCPEG5M7Z7TRGZLROY6IN4UEBUT3EOUHHUSWPA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462759203 (314h24m55.290467608s) [noise=true , address=145.79.227.9:4366 , piece_id=6S2TYRKGTJFMYDULI362X76OD75YW5BPC7UC52MMJFGQO2VLQ67Q, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.462974952 (314h24m55.290683351s) [noise=true , address=145.79.227.29:4941 , piece_id=BR55CUAD3YMJY2R44TQGEIOTAQNIPQIEHFHVGXIRV7EZIUDVZ5KA, seg_pos={0 0}, upload=327057]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: expected piece hash; context deadline exceeded; EOF Dec 18 17:52:54.463235869 (314h24m55.290944254s) [seg_pos={0 0}, upload=327057]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 18 17:52:54.492002807 (314h24m55.319711203s) [upload=327057]: Done uploading segments... Dec 18 17:52:54.492048550 (314h24m55.319756942s) [upload=327057]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031560772s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 112.719421ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:30:25 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 112.750474ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 101.215108ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:29:47 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 101.238949ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 110.761314ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:28:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 110.7872ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 129.554807ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:27:30 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 129.584146ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 106.781787ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:26:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 106.849814ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 102.659307ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:25:42 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 102.678392ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 106.915182ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:24:27 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 106.938242ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 117.482674ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:24:00 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 117.509654ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 114.656192ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:23:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 114.676227ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 121.451346ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:21:59 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 121.470046ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 107.972349ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:21:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 107.994228ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 106.270744ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:21:21 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 106.293149ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 117.835975ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:21:04 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 117.867515ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 108.105651ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:19:50 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 108.126931ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 115.110322ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:19:01 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 115.170051ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 112.50431ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/15ea1cb9093c863b FAIL (2025-12-18 01:18:26 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 112.526363ms edge linksharing bgp eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)