Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 528.672251ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.761437ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 387.902242ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:01:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.930777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 489.564407ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:00:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.652152ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 469.411038ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:59:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.43889ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 486.599019ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:58:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 486.621616ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 391.442898ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:58:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 391.482467ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 517.494ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:57:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 517.53507ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 416.96705ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:57:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.012092ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 430.841438ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.885915ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 432.483455ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:55:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.584688ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 496.867803ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:54:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 496.906804ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 510.020816ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:53:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.053789ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 370.818508ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:52:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 370.843656ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 418.595782ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:52:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.62759ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 400.780917ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:51:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.836586ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 489.219598ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:50:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.246484ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 475.160193ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:50:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.189439ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 462.291382ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:49:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.337713ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 429.097301ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:49:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.115273ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.021846215s trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 11:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02187846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 394.429643ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:15:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.477452ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 397.686607ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.714945ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 454.523464ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:14:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.548765ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 496.766891ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:13:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 496.792031ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 447.436145ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:13:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.465407ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 390.058389ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:12:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.083723ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 360.791923ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:11:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.818379ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 403.389167ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:10:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.414696ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 429.702096ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:10:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.728713ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 360.095716ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:09:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.110788ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 417.847238ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:09:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.873188ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 476.424397ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:08:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.516712ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 558.241992ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:08:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 558.274073ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 369.88147ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:06:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 369.906098ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 395.039084ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:06:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 395.111806ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 444.734065ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:05:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.764556ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 924.706101ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 924.741902ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 367.050122ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:04:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.072399ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 446.061971ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:03:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.083346ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 403.289205ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:02:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.323983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 467.746912ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:01:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.794203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 460.28751ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.31609ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 492.019572ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:00:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.116103ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 601.992785ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:00:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 602.035211ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 455.040864ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:59:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.068479ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 437.663443ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.694121ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 398.80055ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:57:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.867482ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 436.817339ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:57:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.845165ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 467.770988ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.825194ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 442.754433ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:55:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.780318ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)