Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 472.960408ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.98392ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 551.070147ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 551.099796ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 555.655035ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 555.701745ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 459.178458ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:46:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.205093ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 447.611525ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.637868ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 500.647726ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 500.846303ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 571.27008ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:44:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.288408ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 453.190492ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.225588ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 553.198103ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:42:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.224736ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 548.744921ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:41:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 548.771778ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 501.594425ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:40:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.640201ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 485.116303ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.232959ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 433.942539ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:39:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.983327ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 452.780038ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:38:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.903645ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 471.444018ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.472575ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 494.58813ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:36:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.617298ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 539.487165ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:35:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 539.513407ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 550.68408ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.769914ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 466.395707ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:34:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.48439ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 606.024316ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:33:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 606.069459ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 652.607184ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:33:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 652.635852ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 477.407392ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:32:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.468675ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 482.1465ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:31:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.175305ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 541.070879ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:30:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 541.1011ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 472.45745ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.483434ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 512.111495ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:29:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 512.137722ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 534.400118ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:27:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.442161ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 480.564276ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:27:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.613619ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 557.604162ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:26:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 557.633738ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 445.72595ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:25:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 445.751282ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 451.310122ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:24:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.362474ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 500.515161ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:23:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 500.547483ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 556.778196ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:22:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 556.911006ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 469.57425ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.599232ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 457.124866ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:21:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.218981ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 459.712737ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:19:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.739446ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 565.081515ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:19:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 565.111471ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 532.130425ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:18:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 532.156974ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 585.03678ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:17:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 585.068557ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 613.144625ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 613.178855ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 543.28076ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:16:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 543.309551ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 575.09198ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:15:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 575.117127ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 456.963082ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:14:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.993534ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 510.002903ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:13:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.084148ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 518.504692ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:13:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.531169ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 536.866185ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:12:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.894272ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 439.086354ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:11:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.113937ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 478.882864ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:10:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.903988ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 491.358146ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:09:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.384825ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.03465301s trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:09:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.03468269s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)