Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 467.746912ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:01:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.794203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 460.28751ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.31609ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 492.019572ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:00:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.116103ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 601.992785ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 11:00:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 602.035211ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 455.040864ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:59:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.068479ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 437.663443ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.694121ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 398.80055ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:57:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.867482ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 436.817339ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:57:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.845165ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 467.770988ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.825194ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 442.754433ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:55:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.780318ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 474.341697ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:55:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.368398ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 449.515229ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:54:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.544082ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 461.453975ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:53:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.487898ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 367.210648ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:53:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.312418ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 357.961667ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:52:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 357.982334ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 468.674305ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 468.773637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 525.320962ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:51:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.33717ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 356.395016ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:50:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 356.423025ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 440.399997ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:50:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.429542ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 396.270189ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:49:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 396.3007ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 555.285878ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:49:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 555.313707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 453.064751ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:47:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.094304ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 394.536389ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:47:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.566142ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 378.787334ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:46:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.817622ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 454.366467ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:46:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.396999ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 463.444644ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.474601ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 434.052793ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:43:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.100498ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 339.21439ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:43:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 339.247244ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 450.89521ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:42:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.92137ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 466.543729ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:41:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.573612ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 424.82361ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:40:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 424.923047ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 467.323357ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:39:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.352894ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 487.406079ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:38:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.452903ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 430.937203ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:37:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.981387ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 458.762949ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:37:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.796063ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 360.345262ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:36:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.373854ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 431.565681ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:35:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.594845ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 444.813864ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:35:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.844183ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 433.134195ms trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:35:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.158941ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.022945075s trace: https://jaeger.dev.storj.io/trace/2b0803a4d46e222b FAIL (2025-12-09 10:34:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:34:10.740187973 (91h6m11.567896374s) [upload=94696]: Starting upload Dec 9 10:34:10.740414203 (91h6m11.568122607s) [upload=94696]: Done starting upload Dec 9 10:34:10.740559628 (91h6m11.568268007s) [upload=94696]: Uploading segments... Dec 9 10:34:10.741146157 (91h6m11.568854549s) [upload=94696]: Got next segment. Inline: false Dec 9 10:34:10.741286884 (91h6m11.568995286s) [upload=94696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:34:10.741294803 (91h6m11.569003202s) [upload=94696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:34:10.768095096 (91h6m11.595803479s) [seg_pos={0 0}, upload=94696]: Begin upload segment... Dec 9 10:34:10.768148580 (91h6m11.595856972s) [seg_pos={0 0}, upload=94696]: Done begin upload segment. Dec 9 10:34:10.768191138 (91h6m11.595899533s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.768212210 (91h6m11.595920613s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769145272 (91h6m11.596853696s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.769566860 (91h6m11.597275263s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770233084 (91h6m11.597941486s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770270212 (91h6m11.597978615s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.770770489 (91h6m11.598478891s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771251854 (91h6m11.598960258s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.771453080 (91h6m11.599161479s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772454426 (91h6m11.600162829s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.772807579 (91h6m11.600515976s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773346278 (91h6m11.601054669s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773549513 (91h6m11.60125791s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.773877546 (91h6m11.60158594s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774307105 (91h6m11.602015499s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.774387600 (91h6m11.602096003s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Uploading piece... Dec 9 10:34:10.817923537 (91h6m11.645631937s) [noise=true , address=77.90.31.16:4734 , piece_id=UOOGCY4E7KBEUTH76Z7Q4LI3FC5CVJXYZ55MYIRH3I2G36LY2GCA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.820445433 (91h6m11.64815383s) [noise=true , address=77.90.31.15:4698 , piece_id=RFUMMNQIXIVZQZEAATHR3LYV673EWROIWVQCJVBB57XSMVNGUPCQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.821559870 (91h6m11.649268267s) [noise=true , address=145.79.227.30:4763 , piece_id=YVTNSSXTLBQ3GPZ55XVFKAVQVPY5NSDUGOZ6Y3F4TF34KAIEY7QQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 9 10:34:10.865665685 (91h6m11.693374076s) [upload=94696]: Next returned nil segment Dec 9 10:34:10.865679332 (91h6m11.693387729s) [upload=94696]: Waiting for error group managing segments... Dec 9 10:34:10.885051939 (91h6m11.712760332s) [noise=true , address=145.79.227.25:4617 , piece_id=PY2MXA4OLSBEUJFAZLPD374QYRUN2WNOUFKBKRTYJTK2L4NIEB4Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.885824640 (91h6m11.713533032s) [noise=true , address=145.79.227.43:4159 , piece_id=I6CJD5UBFJE3VKPYUWOT72JDZHO7FXXPFTIXPZUNGU72DMYYRGYA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.887968736 (91h6m11.715677132s) [noise=true , address=145.79.227.23:4546 , piece_id=OCZVRIGQNDGSYD2UHBOUHPF7M7VK6YPXJZQI422S3KTHBRUQBZXA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.888662459 (91h6m11.716370851s) [noise=true , address=145.79.227.23:4547 , piece_id=WV5CDEDOBFH6HNZO3VXRGSHPUKWKVXHIZZZ3O74IWMQL57TU64NQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.889187765 (91h6m11.716896157s) [noise=true , address=145.79.227.42:4123 , piece_id=JSAN4T4U6FRRM7XZXUYBIJFMOKB7CHLQ6IJOIGI4RKDFNHATTJSA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891714198 (91h6m11.719422595s) [noise=true , address=145.79.227.24:4583 , piece_id=5J6FU7SRIP7OM6HLUX23XFXQZJP3T247RUDB6BNYTOO2TOQF3KHQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.891752378 (91h6m11.719460769s) [noise=true , address=145.79.227.42:4122 , piece_id=PDRF5RNII6PLDJNP6QZWTKT7QCDZ5O7KLQH5N7XV72OBOBUTD25Q, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.893650254 (91h6m11.721358649s) [noise=true , address=145.79.227.11:4731 , piece_id=UVR4LXTVDFM63HCVSSDYC7CFJCRIPJTFCUIP77OA6XXAWYPSK3EA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.894717866 (91h6m11.72242626s) [noise=true , address=145.79.227.39:4015 , piece_id=UEEPKO65KKJGBEOKDBQ5NWXXRVM7NFQHXFIPV3QH3PZ5IXTH62RA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.895849587 (91h6m11.723557979s) [noise=true , address=145.79.227.31:4798 , piece_id=4ZIDV4Y2EK7JWULKZBT75RO5Z2AKXOA3VZRTEVE63DQYWEVDKYRA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:10.899311983 (91h6m11.727020383s) [noise=true , address=145.79.227.29:4941 , piece_id=BI5KB7FF3ZXRXOML5MZ4VF6YTT4FIW4OSGKHY75LMBDMOWOS7OJA, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:11.004857956 (91h6m11.832566361s) [noise=true , address=77.90.31.17:4772 , piece_id=FJVJ4NPTPTEG6TQD2SHZDN5JCVU4SDCIMYMULJBAFIOOLJF44HWQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: <nil> Dec 9 10:34:40.705653466 (91h6m41.533361867s) [noise=true , address=77.90.31.19:4843 , piece_id=SRLXSLUISBN5XXEOIT2NTMGKBHVWQTHVBTCYBAWTWLJWJBGRSBPQ, seg_pos={0 0}, upload=94696]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 10:34:40.705699272 (91h6m41.533407649s) [seg_pos={0 0}, upload=94696]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:34:40.725964601 (91h6m41.553672992s) [upload=94696]: Done uploading segments... Dec 9 10:34:40.726001121 (91h6m41.553709512s) [upload=94696]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.022964919s core uplink us-select-1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 808.549061ms target: https://ap1.storj.io/login trace: https://jaeger.dev.storj.io/trace/60095d0fd80412a1 FAIL (2025-12-09 09:11:37 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 808.572289ms core satellite ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 808.703967ms target: https://ap1.storj.io/login trace: https://jaeger.dev.storj.io/trace/60095d0fd80412a1 FAIL (2025-12-09 09:10:56 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 808.728521ms core satellite ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 761.445724ms target: https://ap1.storj.io/login trace: https://jaeger.dev.storj.io/trace/60095d0fd80412a1 FAIL (2025-12-09 09:10:04 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 761.537173ms core satellite ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 736.473343ms target: https://ap1.storj.io/login trace: https://jaeger.dev.storj.io/trace/60095d0fd80412a1 FAIL (2025-12-09 09:09:15 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 736.549989ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.585573598s trace: https://jaeger.dev.storj.io/trace/1dcc6772c2f7dbab FAIL (2025-12-09 09:09:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.585602895s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.765005068s trace: https://jaeger.dev.storj.io/trace/1dcc6772c2f7dbab FAIL (2025-12-09 09:08:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.765040088s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 792.111097ms target: https://ap1.storj.io/login trace: https://jaeger.dev.storj.io/trace/60095d0fd80412a1 FAIL (2025-12-09 09:08:02 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 792.153977ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 4.402617708s trace: https://jaeger.dev.storj.io/trace/1dcc6772c2f7dbab FAIL (2025-12-09 09:07:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.402646247s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.534917285s trace: https://jaeger.dev.storj.io/trace/1dcc6772c2f7dbab FAIL (2025-12-09 09:07:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.534956568s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.728486978s trace: https://jaeger.dev.storj.io/trace/1dcc6772c2f7dbab FAIL (2025-12-09 09:06:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.728512004s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)