Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 11.035222053s trace: https://jaeger.dev.storj.io/trace/69068312b89d7c06 FAIL (2025-12-21 16:09:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.035252329s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 511.75523ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.780768ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 683.015221ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:35:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 683.03418ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 479.891464ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:34:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.91751ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 456.823551ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:33:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.865713ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 442.954369ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:32:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.99766ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 441.001811ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:31:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.036952ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 432.524763ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:30:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.560859ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 461.217285ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:29:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.23851ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 446.0083ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:28:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.121042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 416.925927ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:28:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.959156ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 399.982351ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:27:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.010068ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 407.038838ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:27:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.066201ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 430.80865ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:25:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.836302ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 445.647708ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:24:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 445.677969ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 504.214023ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:24:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 504.240501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 523.869511ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:23:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.895072ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 529.023597ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:22:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.145675ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 450.229959ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:21:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.266816ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 453.099952ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:20:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.124317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 442.659735ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:19:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.688998ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 532.221049ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:19:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 532.273663ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 487.435722ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:18:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.464066ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 581.676923ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 581.706707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 427.658163ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:17:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.68745ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 492.064023ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:17:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.092693ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 405.817346ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:16:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.844145ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 419.114083ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:16:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.139989ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 419.662943ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:15:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.691923ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 400.024727ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.141554ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 511.193793ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:14:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.235718ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 448.821012ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:14:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.848413ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 418.120187ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:13:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.146198ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 571.194743ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:12:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.280628ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 521.5594ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:11:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.590941ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 512.621426ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:11:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 512.649088ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 431.861553ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:10:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.892198ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 528.047121ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:09:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.080574ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 503.007986ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.040963ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 493.04944ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:08:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.078205ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 403.89911ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:07:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.931511ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 450.133757ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:06:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.160911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 422.4255ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.456973ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 425.424344ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:04:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.462169ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 566.108085ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:03:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 566.135096ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 577.223717ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:02:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 577.332797ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 399.186817ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:01:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.217675ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 407.397443ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 15:00:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.430886ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 487.422706ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 14:59:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.448569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 446.388915ms trace: https://jaeger.dev.storj.io/trace/60013b57840b8670 FAIL (2025-12-21 14:58:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 21 14:48:35.208910548 (383h20m36.036618945s) [upload=398891]: Starting upload Dec 21 14:48:35.208965530 (383h20m36.036673931s) [upload=398891]: Done starting upload Dec 21 14:48:35.209225644 (383h20m36.036934035s) [upload=398891]: Uploading segments... Dec 21 14:48:35.209267032 (383h20m36.036975429s) [upload=398891]: Got next segment. Inline: false Dec 21 14:48:35.209298408 (383h20m36.037006802s) [upload=398891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:48:35.209303116 (383h20m36.037011507s) [upload=398891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:48:35.246827936 (383h20m36.074536327s) [seg_pos={0 0}, upload=398891]: Begin upload segment... Dec 21 14:48:35.246877959 (383h20m36.07458635s) [seg_pos={0 0}, upload=398891]: Done begin upload segment. Dec 21 14:48:35.246911319 (383h20m36.07461972s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247218213 (383h20m36.074926616s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.247970233 (383h20m36.075678627s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248493316 (383h20m36.07620171s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248639760 (383h20m36.076348162s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.248914690 (383h20m36.076623081s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249310412 (383h20m36.077018805s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249362388 (383h20m36.077070786s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249713170 (383h20m36.077421561s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.249995699 (383h20m36.077704101s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250100446 (383h20m36.077808838s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250512045 (383h20m36.07822044s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250557413 (383h20m36.078265815s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.250860348 (383h20m36.078568741s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251217843 (383h20m36.078926248s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.251866331 (383h20m36.079574734s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Uploading piece... Dec 21 14:48:35.309661017 (383h20m36.137369398s) [upload=398891]: Next returned nil segment Dec 21 14:48:35.309673439 (383h20m36.137381821s) [upload=398891]: Waiting for error group managing segments... Dec 21 14:48:35.319714746 (383h20m36.147423137s) [noise=true , address=145.79.227.49:4338 , piece_id=5CUS5TMAV2HENROSH3N5JRS2EL2GDQLHHKBYBJHR2ZBQIOJRH6LA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.320180766 (383h20m36.147889157s) [noise=true , address=145.79.227.27:4690 , piece_id=XWSZU6T5LLH6MFJBAHZTVBPHSF73J3H2BWJMAZBFTV6U66U6MFQA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.323625111 (383h20m36.151333503s) [noise=true , address=145.79.227.36:4517 , piece_id=JVRU3RUDTM6OW2O5YC6L7CEN6KHE2VAYZAWWFBTFA7RKSY5AI46Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324433197 (383h20m36.152141589s) [noise=true , address=145.79.227.50:4373 , piece_id=6C2F3J5RTI2UDMW2FQ2CZSG26JLAFGF6BCFVEDGLVCS5ZJPEMNWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324849789 (383h20m36.152558182s) [noise=true , address=145.79.227.45:4230 , piece_id=ZRODQSPZZRJUQTXJ7W7NCQPMS3WHGSEGCE57MO745QRLAODTPL6A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.324917589 (383h20m36.15262598s) [noise=true , address=45.91.116.8:4046 , piece_id=HTU63D6RE6IUYFVOBBTXC2UQQFIZMFZZKPXWLQT5VJV3SWJUUNOQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.325357173 (383h20m36.153065567s) [noise=true , address=145.79.227.38:4978 , piece_id=63SFM77QWCYIQB55LWAV3L7PZLZ6WPSFIVGKKINRRYLCXZ6OXHUA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.326179304 (383h20m36.153887695s) [noise=true , address=145.79.227.48:4302 , piece_id=VKUG23TADUHL2M3TJARK2HRRH3MTIAGGSCXDPELHOSXXNGCEN6JQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.334132619 (383h20m36.161841011s) [noise=true , address=45.91.116.10:4081 , piece_id=MVXPEY4IO4UMPU3CTBLGT7R7EX3JTHGT4NA55R67Z6LZ7O2Q2NWQ, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.338292172 (383h20m36.166000563s) [noise=true , address=45.91.116.12:4227 , piece_id=NRA6HLLT3POBMULHMR2DHB35FGEJAEZNQXOQPKPPGNIPHJUEP5GA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:35.375591835 (383h20m36.203300228s) [noise=true , address=77.90.31.10:4520 , piece_id=KFGGQAJVLP67X52MOBCG7KPLGL6N5GKVZ4KCWYK52WDGRC6KHJSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:48:37.850982304 (383h20m38.678690695s) [noise=true , address=145.79.227.39:4014 , piece_id=RNMLJBJ66X6FWCDXQWDLFSRPOVI6N6RMXEICE7FCYILLIETP6Z2Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: <nil> Dec 21 14:49:05.172572805 (383h21m6.000281206s) [noise=true , address=145.79.227.42:4121 , piece_id=X3JVYQLT4JLZQWIIB7RJX3LRDGTHE3KNO6PIMEPW24OWFOYBHGSA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172616657 (383h21m6.000325081s) [noise=true , address=145.79.227.32:4836 , piece_id=3L5SD7OZENYQOIAE7BDWFGGVN2TZGNNNJK6ZG6U5BLFBSUKTY27Q, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172750719 (383h21m6.000459119s) [noise=true , address=145.79.227.10:4402 , piece_id=EKN65OKLPMDNU37Y4CSKPZ5MSDQD47EECI7YFA63Z63N3XWNGH4A, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172900971 (383h21m6.000609365s) [noise=true , address=45.91.116.4:4940 , piece_id=5AI4QVZRF6EHIW6CCPFNNRYNDP6KX3ZT7X62YCZRQY7AXYXZ4WIA, seg_pos={0 0}, upload=398891]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 21 14:49:05.172932795 (383h21m6.000641187s) [seg_pos={0 0}, upload=398891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 21 14:49:05.203212563 (383h21m6.030920954s) [upload=398891]: Done uploading segments... Dec 21 14:49:05.203251269 (383h21m6.03095966s) [upload=398891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.421455ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)